tttjjj/elt_bench huggingface dataset:
I believe that the question should be rephrased can there be finite ***levels*** (or countably infinite) for a categorical ***variable***.  Categorical *data* can include infinite values - there's no reason to assume that an observation has a given cardinality.
Error in randomForest.default(m, y, ...) : 
  NA/NaN/Inf in foreign function call (arg 1)
In addition: Warning messages:
1: In randomForest.default(m, y, ...) :
  The response has five or fewer unique values.  Are you sure you want to do regression?
2: In data.matrix(x) : NAs introduced by coercion
I think there's a typo and you invert `se` and `sd`; BTW your `mse` (I assume it is mean square error) is actually the SS.
Thanks for the answer. I had thought that the simple methods had inherent weaknesses, but was also concerned that stochastic methods may suffer from lack of consistency between analyses. Need to look into it further.
Maybe your V(E(X|X1..Xn)) is not correctly calculated. The answer should be the same.
It's a little hard to understand the structure of your experiment and data. It sounds like both species and treatment were manipulated between experimental units (i.e., each unit is in only one "tub"), is that correct? How are the blocks different from one another? You said each one contains all spp-x-trt combinations, so is it just different units in each block? Are you interested in change over time, or is that just multiple observations?
+1 for the linking $\\chi^{2}$. This tutorial is awesome.
Let $G \\sim {\\cal N}(\\mu, \\sigma^2)$ ;-)
Multiplication part is totally wrong
I edited the question to add more details. I hope now it is clearer what I want to do.
So really the spatial element of this problem is more of a nuisance rather than the story. If I were interested in the value at a specific location I agree that the kriging prediction would be error would be the ticket. If one were to view the problem as more of a weighted least-squares problem though, how would the prediction interval formula change?
I agree with you, that these 7 data-points are almost meaning-less if there is no context around them. But GaBorgulya, I'm not very concerned about the information around these sample data-points, but what I'm concerned about is- how these data-points should be treated in-general. And as I quoted in few of my comments, I found these hard-limits at:  
http://bit.ly/ibQhWo, 
http://bit.ly/fWhRTD

I'm not saying what he is doing is very optimal or good solution, but what I'm saying is he came up with interesting conclusions, which are really making some sense for me.
`since change-X is a [linear] function of pre-X and post-X` they three cannot be linear predictors simultaneously. Remove one of them as a redundant term.
can you explain more regarding what you mean by 'centered' on the last day of the study? why not just pump in day as 0 through number of days on the last day of the study? Also- in your summary table- where are the interaction effects?
The quick and easy answer is to add the values. I would probably do that if I were you. It is not perfect but it provides very meaningful results most of the time and a majority of researchers do it this way.
@chl: Yessir! The OP asks for $E(N)$ where $N\\ge1$ is the date of the first success in a series of independent trials, each with probability $p=1/27$ of success. The event $[N\\ge n+1]$ corresponds to the $n$ first trials being unsuccessful, hence (by independence) $P(N\\ge n+1)=(1-p)^n$. Now, for every nonnegative integer valued $N$, $E(N)$ is the sum of $P(N\\ge n+1)$ from $n=0$ to $+\\infty$. For every complex number $a$ such that $|a|<1$, the sum of $a^n$ from $n=0$ to $+\\infty$ is $1/(1-a)$. For $a=1-p$, one gets $E(N)=1/p=27$.
Err, actually, not.  The wikipedia page labels the features $F_1\\ldots F_n$.
200 & 400 is not too unbalanced, I think, especially since you have so much data. However, your power will not be as great as if they were 300 & 300. MW is less powerful than t if the data are normal, but more powerful if they're not. You could always report both medians & means, or whatever you like.
+1 I think this is a great question: it cogently challenges accepted wisdom, forcing us to think harder about why certain practices are correct and good. But, to narrow the scope a little, I would like to ask about your situation. (1) Are you certain the relationship between the dependent and all independent variables is linear? (2) Are you certain of your independent variables--that is, should they all be included or not?  After all, you mention "overfitting": bear in mind, then, that with $p$ possible variables you have $2^{p+1}-1$ possible models to fit. That might require a lot of data!
See [here](https://stat.ethz.ch/pipermail/r-help/2006-September/112779.html).  Looks like `summary()` will do it in `lmer`
Means and variances have a formal definition for any distribution.  The Cauchy is a distribution where they don't exist because the integrals defining them are infinite.  But bimodal and many nonnormal distributions can have finite means and variances. As long as you have finite means variances and covariances the Pearson correlation is defined.  Sometimes the interpretation of these parameters is difficult for non-normal distributions but they are very natural for normal distributions.  The variance can always be helpful in terms of the information it gives in the Chebyshev inequality.
@dsimmie Forgot the (a) to ensure you actually see the comments above.
An answer from the man himself - Thanks! Will definitely have a look :)
Are we talking about different things? The binomial distribution is discrete, a c.i. would be "for $p = 0.8$, in 94 % of the repetitions we observe $k \\in \\{3, 4, 5\\}$ successes in $n = 5$ tests". But I understood that we are to estimate $p$ for already observed $n$ and $k$. E.g. $p$ given that $k = 4$ out of $n = 5$ tests were successes. So I'm talking about $Pr (p | n = 5, k = 4)$, $p \\in [0, 1]$. This is *not* the binomial distribution $Pr (k | n, p)$ but that of proportion $p$ (I don't know its name). Please help me to understand why there is no density for this distribution?
Anyway, read this first http://en.wikipedia.org/wiki/Poisson_process
Neural networks went out of fashion for a while in the 90s - 2005 because they are hard to train and other techniques like SVMs beat them on some problems. Now people have figured out better methods for training deep neural networks, requiring far fewer problem-specific tweaks. You can use the same pretraining whether you want a neural network to identify whose handwriting it is or if you want to decipher the handwriting, and the same pretraining methods work on very different problems. Neural networks are back in fashion and have been outperforming other methods, and not just in contests.
A very nice package! Will you be supporting higher-order Markov Chains?
By variogram, you mean variogram **model**?
Hi, welcome to the site. Please do not write all caps. Also, you should probably add more information to the question (what are you trying to do, what errors you get).
Can you say more about what "they are spread uniformly in time and in space" means? I can think of few different things it might mean.
Ah, in that case you can definitely ignore ordinal logistic regression, as that's for an ordinal *outcome* variable.
Hmm how can you characterize the sampling distribution with bootstrap methodes? Would you be so kind to describe this for an amateur? Thanks alot
This is a time series analysis. Word of warning, there is more noise in economic data than trend. You will probably find some sort of "pattern" that, if checked on another dataset, will not explain the trend.
It performs better of your sample sizes are equal (if my memory serves me right), but the results are valid if they are not.
In the past, I have seen reports based on convenience samples say "x% of respondents", and then watched people interpret that to mean "x% of the population".
This is a fairly specialized field. I'd consider a focused literature search (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3796884/) before posting such a broad question.
I think in this context the vertical bars can read as "given that". So the first line would mean $v_{i}$ **given that** $z_{i} = k,\\mu_{k}\\sim N(\\mu_{k},\\sigma^{2})$. See also [this list](http://en.wikipedia.org/wiki/Vertical_bar#Mathematics) on Wikipedia.
@user, you do not need to add to a supermodel anything, just supply `jtest` or `coxtest` with non-nested fits from Step1. Information criterion for non-nested will be a nice guide to what model is more statistically suitable (parsimonious), but for the hypothesis testing I would just go for any $LR$ (actually the log likelihood is a part of any info-criterion) tests myself. Conclusions will be somewhat close, but since there are two deterministically given penalty functions it is a bit tricky to compare them statistically.
@f1r3br4nd In a 2x2x2 design, there is only one unique A*B*C interaction contrast (like there is only one in a 2x2 case). The coefficients in an A*B*C interaction-contrast have to sum to zero over rows (A), columns (B), and planes (C) in the "design cube". If the order of cells in the associated one-way design is `a1.b1.c1, a2.b1.c1, a1.b2.c1, a2.b2.c1, a1.b1.c2, a2.b1.c2, a1.b2.c2, a2.b2.c2`, then the coefficients are `c(1, -1, -1, 1, -1, 1, 1, -1)`. If you have more than two groups in your factors, all contrasts following the sum-to-zero rule are 3-way interaction contrasts.
If you have 2 lakh (i.e. 2 hundred thousand) cases where the number of dependents is unknown, out of about 1 million, then you may be  able to include the fact that you don't know the number of dependents as a factor in the analysis.
That's amazing - thanks for your help
A generic way is to perform a binary search within a list of the cumulative probabilities, which in this example would be $(0,0.04,0.54,1.0)$.  On the average this takes $\\log(n)/2$ probes per generation event.  If no probability is extremely small, you can get $O(1)$ performance by creating a vector of equally-spaced values in $[0,1]$ and (in a precomputation stage) assigning an outcome to each value.  E.g., in this example you might create the vector $(1,1,1,1,2,\\ldots,2,3,\\ldots,3)$ (with $50$ 2's and $46$ 3's).  Generate a uniform, multiply by 100, and index into this vector: done.
Great article - thanks!
I suggest, for posterity's sake, that we rename MAR to MAAR ("Missing (Adjusted) At Random"), which would be the same length as MCAR and MNAR, and would also be pronounced the same as MAR when spoken as an acronym.
I see the problem; Welfords obscures the problem with this method: the online estimate of the mean is being used instead of the final estimate of the mean. While Welford's method is exact (up to roundoff) for variance, this method is not. The problem is *not* due to the `sqrt` imprecision. It is because it uses the running mean estimate. To see when this will break, try `xs <- sort(rnorm(n.testitems))`  When I try this with your code (after fixing it to return `a.dev / n`), I get relative errors on the order of 9%-16%. So this method is not permutation invariant, which could cause havoc...
@gung, I have tried to edit my question to make clear that I want to create a data set that corresponds to the expectations of the test/model.  I am not looking for something for a simulation study or for a robustness study, but to provide a dry-run of an experimental design.
Weight is a ratio and an interval variable. The point is, many interval variables take on values that are real numbers, and are not restricted to integers (or anything with a 1-1 correspondence with integers, such as when you shave off a certain number of decimal places). In that case, the interval variable is not nominal.
It's not clear what your question is, because you say you have been doing regression. This problem presents several difficulties, such as reporting data in two-year intervals (which conflicts with the statement that these are "annual installations") and the vague and confusing note at the end.  Where exactly are you looking for help? With understanding sigma notation? Performing a linear regression? Making sense of this (terrible) homework question?
@Pepper: Yes, you could also fit topic models recursively as you say. That may or may not be more computationally tractable, I'm not sure, but I suspect the resulting topic hierarchies would be considerably inferior in quality. Inferring the hierarchy all at once, as with hLDA (or nHDPs), will provide the optimal global structure, whereas recursively fitting LDA models to construct a tree would be locally optimal, a sort of [greedy algorithm](http://en.wikipedia.org/wiki/Greedy_algorithm). I'm not expert however, so if this point is important you might make it a new question.
This is a great answer. If you want more, I cover somewhat similar territory here: [What if residuals are normally distributed but Y is not?](http://stats.stackexchange.com/questions/12262//33320#33320)
@Procrastinator: Given that the outcome is a non-transmissible disease, it seems reasonable to assume that members of the same household can have differing outcomes.
This is in general an answer for any bootstrap test (or multiple imputation test) in SPSS. If one of the subsets did not converge it will not report the pooled analysis. I think from the OP's question they have dichotomous data, so with only 30 some observations this seems like a real possibility. I'm not sure what to say in response that SPSS reports all of the confidence intervals - maybe it reports the interval with zero length in the case of no variance.
@user28246 try `summary(mdl1)`.
Readers here may be interested in the following general overview of topics related to power analyses via simulation: [Simulation of logistic regression power analysis: designed experiments](http://stats.stackexchange.com/questions/35940/).
Do you want an explanation along the same lines as [here](http://stats.stackexchange.com/a/4287/44764) for Bayesian regression models, or are you looking for a discussion of bias and variance in sampling-theory regression models from a Bayesian perspective? Bias is automatically induced in the Bayesian approach just from using a prior, while the variance is decreased for the same reason.
Here are the weights for the 13 items with each weight between 0-1.  0.63,0.63,0.63,0.60,0.56,0.56,0.53,0.50,0.50,0.44,0.38,0.38,0.38.  If we now assign 50/100 score to each item and calculate the total score, it will be 336.  Note that the user only has 650 points total in the bucket, but can assign 0-100 for any item as long as there are points remaining in the bucket.  If the user assigns 100/100 for each of the top-6 weighted items and 50/100 for the 7th most weighted item, the total score will be 387.5.  My problem is that the spread between 336 and 387.5 is not big enough to detect fraud
Why do you have to check manually? Plus P&R make direct sense only in case of binary classification, how do you want to generalize into many clusters?
For example, rough sets are mentioned as data mining tool in http://www.comp.dit.ie/btierney/BSI/Data%20Mining%20Tool%20Reviews%20March%202011.pdf .
Thanks Gung! I'll look through your reference.
Thank you so much, Peter! I appreciate your input! If I want to find a parsimonious model to have correct prediction results, what strategy should I take? (Using what statistics to compare...)Thank you!!!
Have you looked into Box-Cox? It's been discussed here a lot, and there are also lots of resources on the web.
[This question](http://stats.stackexchange.com/questions/12641/addressing-model-uncertainty?rq=1) and at least the first answer (didn't read the others) have some useful discussion. In short if you believe the true model is one of these possible models then the BIC will give you **some** theoretical justification for choosing your model. As a last resort Bayesian Model Averaging (BMA) would be the most comprehensive way to take model uncertainty into account.. Either way you'll have to do something other than just take the MAP. The specifics of your modeling scenario might help others help you
@whuber Not necessarily, unless we are going so far as to call a group of colleagues, a white board and a marker "automation" (which is typically what I start with for variable selection via Directed Acyclic Graph analysis). The criticism is two-fold, one that it is particularly bad at selecting the right set, and the other is that the researcher should *never* simply abdicate the process to software.
I dont get that result, can you illustrate please?
You should write such a comment as... a comment, not an answer.
For some intuition, note that the likelihood scales with the sample size while the prior does not.
How did you compute the posterior of $\\alpha,\\beta$? Did you use BUGS, or did you code it yourself? Please, show us your code.
Although I did not look into the inequality itself, I believe that $|\\rho| \\leq 1$ is always true since it is a correlation coefficient, which I guess is defined by $\\rho = cov(X_1, X_2)\\big/ \\sqrt{var(X1)}\\sqrt{ var(X2)}$...is your question: "is the inequality true?" or do you question the inequlaity under some assumption regarding $\\rho$?
I've added the 'homework' tag again here, if that's not right, go ahead & change it.
I understand the 1st 2 paragraphs of this answer. On the 3rd paragraph, how would you tell the data points only take limited (whole number?) values if the horizontal lines were not there? It seems to me that this might be a case where the horizontal lines actually help you to work out if the y values are all whole numbers.
@Transcendence: Might I kindly suggest you reread the comments from start to finish, taking special care to read the wording. Three users were unsure of what you were asking and so asked for clarification in one form or another. This is not proof-positive that the question needs refinement, but it's strong evidence. In Xi'an's first response to you (his second overall comment) he specifically indicated the word (*distributed*) he thought was missing. I find your long comment starting with "No. I'm not trying to define..." quite confusing, especially in like of the other comments before that...
@ttnphns [This article](http://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1004&context=pb_pubs) may provide some of the missing details. PA is not a permutation procedure, since you're not sampling from the empirical distribution function, but from multivariate normal data.
I agree with Peter Flom that the problem is definitely more difficult than sketched here if you want to know about a trend, then reading times are necessary (or are they equally spaced).
Did you read some advice for a [similar question](http://stats.stackexchange.com/q/12200/3277)?
@GaëlLaurans, my point is that due to rounding (whether concious by human, or by computer) the concepts of exactly equal and within X% for a sufficiently small X are practically the same.
Why would you do this if you have labels for the data? My understanding is that the label propagation you're referring to is only useful when you don't have labels associated with those observations.
Ok, then think on previous suggestions and let me know the results you obtain.
Sample size of 40 is likely to be underpowered. Given what you found, r=0.26, a sample of 40 and alpha of 5% yield a power of 34.6% which is pretty low compared to the conventional 80%. In order to find an association of r=0.1, given alpha = 5% and power = 80%, you'll need n=617. For r=0.2, n=153; r=0.3, n=67; r=0.4, n=37. So, unless there is a moderate association of r=0.4 or above, you'll have hard time detecting it. [Online calculator](http://www.stattools.net/SSizcorr_Pgm.php) I used.
I would look at quantile normal plots of the distribution of times for each server. If in doubt, I'd run both the t-test and the nonparametric test and see if there are differences.
@richiemorrisroe There are certainly some people who use it for whatever reason - they like it, they dislike manual formatting, whatever. I have however found, hopping between fields, that I run into fields that almost *never* use LaTeX (my own) and fields where it was once asserted "If its going to be published, it must be in LaTex". That conversation ended in me having to go print out a copy of AJE's submission guidelines.
Kevin Murphy's [probabilistic modeling toolkit is nice](http://code.google.com/p/pmtk3/)
Thanks @user27840 for your response. My dataset is approximately balanced (48.47% having label 1, remaining label 0). Are you aware of any research paper which mentions that k-fold cross-validation might give such high accuracy without cross-validation? I am pretty convinced (especially after reading your response) that there is not much overfitting. But the reviewers of my journal where I am using this model think otherwise. So I need a way to convince them and perhaps citing some other work might be worthwhile.
the $x$ in your second question is one row of what you wrote $Y_x$ above.
The master rule is *not XLS*.
Do you mean the cumulative moving average?
(+1) I looked for this one, but cannot found it in the `ape` package!
@Andre I think this remark may be closely related to another one offered by @Michael Lew elsewhere in this thread (http://stats.stackexchange.com/questions/4551/what-are-common-statistical-sins/4567#4567 ).
Hi Pedro, I have made some edits to my question. Can you provide some help here. Thanks!
I want quantile regression because for research purpose.also I use STATA.Also thank the replies for you.Rubber yield take as production.
How many transactions do you observe and how long is your time series? And what is your model?
Thanks for elaborate answer sir. But just to confirm again, i am not squaring x1x2 like you showed above, rather i am squaring the z-score of x1x2. Along with that in the regression model, i include x and x2 also along with certain other control variables. Does it change your viewpoint now?
I was looking for the estimation of the sum of this variables.

I have about 10 regions and 3000 sub-regions. For some additional 300 sub-regions that forms a region I predict the log of spending in a commodity as the log of its population by income range using a OLS model estimated with the 3000 sub-regions data. I have these predictions and their confidence intervals. What I want is the prediction and interval confidence for the total spending of the region, i.e., the sum of the 300 sub-regions. Its distribution would be also useful.
I do appreciate the constraints (and should have been explicit about that). In general, though, a good framework for resolving questions like this is to take an appropriate model as your point of departure. If you try to reason your way to a solution using an inappropriate approach and invalid model (to satisfy a client), you will only compound the errors and be unable to reach any clearly defensible solution. What you might consider now is how the Deming regression varies from inverse regression, as well as how Deming regression could be adapted to accommodate multiple variance components.
If you use R, see the **ppoints** function.
@Momo - I mean that viewing the ROC curve did not help understand model performance very well, and even more  the ROC curve did not lead to any insight or to good behavior.  Generalized $R^2$ measures are simple translations of likelihood ratio $\\chi^2$ statistics and so are powerful.  Michael - I'm not really interested in sensitivity or specificity because of the backwards time ordering.  I want to know that a predicted risk of 0.2 means that the actual risk is very close to 0.2, and I want decent predictive discrimination as measured by a rank correlation measure or generalized $R^2$.
Looks good but I don't know how to install that 'aplpack' to my computer (sage 5.8, r-base 2.15.1-5ubuntu1).
I'd be impressed if any exist. But just think of technical terms that you would expect to appear in such lectures and not in industry or introductory lectures. (But are lecture videos indexed by words in them???)
One algorithm I have found is Halite http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5989805 , which seems to match most of these criteria