tttjjj/elt_bench huggingface dataset:
+1 for the point of overfitting.  Algorithms such as Naive Bayes and linear discriminant analysis often work very well, even if you know the underlying model is incorrect (e.g. spam filtering), simply because less data is needed to estimate the parameters.
adding "time" is a very bad practice as there is an implicit assumption of 1 and only 1 trend that starts at the beginning . If there is no "trend" for the first k periods , the correct series to use would be 0,0,0,0,,,,k,1,2,3,4,5..... additionally there may be level shifts (intercept changes) that shouldn't be confused with trends.
@ostrodmit `The euclidean distance, which is in RBF, is completely uninformative even in 7-8 dimensions.` On what do you base this assertion?
@cardinal, you're right about that. I edited the answer to note that the intercept is required.
It seems that the correct code is `lmer(Y ~ X1 + X2 + X3 + X1:X4 + (1 + X1 + X2|level2) + (X1 + X3 - 1|level3), data=mydataset)`.
@Néstor, +1 to this answer but re: your last comment, I think there is a distinction between big-O and [little-o](http://en.wikipedia.org/wiki/Big_O_notation#Little-o_notation) notation.
Exactly, yes. Different SNR alone (with the same guessed parameters) is more than enough to get different *estimates* (i.e., realizations of the *estimator*). Can I post this last comments as an answer?
This all seems to assume the months are uncorrelated - have you made that assumption explicit anywhere? Also, why do we need to bring in the normal distribution? If we're only talking about variance then that seems unnecessary - for example, see my answer [here](http://stats.stackexchange.com/questions/31177/does-the-variance-of-a-sum-equal-the-sum-of-the-variances/31181#31181)
I don't think there's a single best answer, so this has probably to be made CW. Moreover, I think you need to clarify what you intend to do, and what's your background.
At this point, with your edits, it's questionable whether this is a statistics question anymore (might be programming). If you really wanted to know how to test Benford's law you should have asked that, and preferrably state what language you want to program it in, where the questionable data are, how you want to test, etc. With so many edits and revisions you should just accept an answer to this question, pull your revision, and ask a new question. In it you could refer to this question as an example of the attempt you have made at testing.
@Gritti: Actually, could please you elaborate on why you think that--from an Economics perspective--the latter would be correct?
Following on from Charlie, you *might* be in the happy situation where the non-random selection process is the same for each block.  Then your original question 'is the X Y relationship different between blocks?' is actually addressable by the usual methods despite the fact that each method will be arbitrarily wrong about what that relationship actually is...
Actually I didn't say that but the person who edited my question made it look like it. Anyways, thanks for your answers and both cases are interesting.
Why do you think it looks wrong?
Why not look at a graph or tables or definition of $t$?  You will see the answer immediately.  It becomes obvious that something would be wrong using $t_\\alpha$ because for $\\alpha\\lt 1/2,$ $t_\\alpha \\lt 0,$ which makes it useless in comparison to the explicitly non-negative $|T|$.  That also makes it clear that something is wrong in your characterizations of t-tests.
A second reference for the definition of recapture rate. http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB4QFjAA&url=http%3A%2F%2Fwww.nature.com%2Fnature%2Fjournal%2Fv432%2Fn7016%2Fextref%2Fnature03039-s1.doc&ei=VhJzULbqEIrKrAeS2IDACQ&usg=AFQjCNHR-bPJ3xYBZpnSKgVUedl4kMjs4A
@Stéphane Your answers were good and I get it now. My remaining issue is that I am still not positive that it makes sense to assume within variation is related to between variation when analyzing a single data set. In the long run sure, but I am not sure it is useful to me to care about "the long run" as a researcher.
@Glen That's true, however, I did think there must have been a pre built package to do this task quickly, as it's a very standard part of regression analysis. To be honest, I'm surprised it's not built into the R language's lm function from the start!
I use KDE for samples in general. But sometimes it happens to see a sample where all numbers are equal or that most is.
For example, for the vector $\\vec{x} (70,70,70,70,100)$, $IQR = 0$, and on Silvermans' rule, the bandwidth would be 0 due to $\\min(s, IQR)$.
Of possible interest: [Multinomial logit](http://en.wikipedia.org/wiki/Multinomial_logit).
See the answers [here](stats.stackexchange.com/questions/6652/what-precisely-is-a-confidence-interval) which may help clarify the concepts.
See also answers and comments at http://stats.stackexchange.com/questions/60725/validity-of-normality-assumption-in-the-case-of-multiple-independent-data-sets-w However, the implication seems to be that you really do have only 6 measurements, not lots of small samples.
Are you looking for a list of all possible transformations on the predictors? That might be too much asking...
aaand I thought I was the only reputation hunter :-)
Thanks for pointing in a direction.  Unfortunately news group posts are far from compelling evidence when discussing these issues in published work.  The general primer on sphericity is interesting, and points out why Bonferroni-stype p-value adjustments are still in common use.  Unfortunately, there isn't really an indictment of the conservative nature of that approach.
@Scortchi: is your last centred formula correct? There is an equality sign missing, and also, what is $\\sigma$? Apart from that, what exactly do you mean by "uncorrelated estimates for $\\beta_1$ & $\\beta_2$"? Each $\\beta$ is a number; two numbers cannot be correlated or not; what am I missing?
Thank you! I shall preserve this link because I used it to prove the above.
@cardinal Usually the densities will only cross in one place (where the right hand tail of one meets the left hand tail of the other). But say the one on the right has a very large variance so that the density is very spread out then it can cross at both the right hand tail (dropping below as you move left and then the lefthand tail crossing above. I didn't say that they couldn't intersect. I just meant that the intersection point need not necessarily be the point you want to define as the separation point.
@MichaelChernick I've updated the question with algorithm's pseudocode. I hope it is now possible to answer my question without reading the article.
+1 thanks Glen_b. I now have a similar problem, but with random effect: http://stats.stackexchange.com/questions/90702/how-to-fit-log-linear-poisson-autoregressive-mixed-model - would you know the answer? I've put 200 bounty on it.
Are the $x_i$ and $y_i$ independent? If so, the inner product is a *subexponential* random variable in the sense of R. Vershynin. See his recent expository introduction to nonasymptotic random matrix theory for relevant bounds of the type you desire.
thanks @ a11msp
Peter, that's interesting. In that case we may be delighting a lot of people! Though I think it's easier to click a single button on a web page compared to filling in a response card which takes a bit more commitment and may filter out those less determined to express their views.

What would concern me a little is if the response rates are biased towards certain pages because of this. However, my main goal at the moment is to identify the pages which are most in need of attention, so hopefully this is still covered even if some generally satisfied users aren't responding.
I'd buy a street directory, turn to the index, and (if it's just too many to count the entries) measure the number of column-inches (/cm) of streets listed. Then I'd subtract the distance taken up by any headings, whitespace or other entries that don't constitute a new street. Then I'd count how many streets are listed in say 4 inches (or 10cm). Then I'd multiply by the total column inches to get an estimated count. Then I'd check for any double counting (say where streets are listed twice). Or, I'd try phoning the city administration, since they might already know the number.
@Raghvendra if you're using a standard classification of occupations, which has a hierarchy built in, can you go from (say) 5-digit occupation codes to 4-digit ones, using the hierarchy as the method for aggregating groups?
@Maisu if this answered your question, you can accept it as the correct answer by clicking the check mark under the number of votes for this answer
hm, if you know how to deal with numbers, then get them by multiplying true mean by 0.5%.
. My canonical examples are survival analysis and equivalence testing in medicine (particularly pharmaceutical research) is not so important to MBA students but time series analysis is important to MBA students but not really very important in medicine (I don't include longitudinal analysis under the category of time series even though it involves repeated measurement over time).  Longitudinal data analysis is something that people in medicine do need to know something about but may not be important in some other disciplines.
(that is, unless I introduce a hyperparameter to make the xx(other) less desirable, the system will choose it too early)
Thanks @chi ! It worked great.
Much like causality, it is what you define it to be. See a possible definition here: http://en.wikipedia.org/wiki/Algorithmically_random_sequence
It's the distribution of the difference that matters, not the distribution of the variables.
Thank you probabilityislogic.
Thanks a lot, I will try it tomorrow.
@Vincent many classifiers output negative scores. SVM is a popular example, where the predicted class label $y \\in \\{0,1\\}$ is then computed as `sign(score)`. A score can have any value and is certainly not necessarily a probability.
+1, and I took a liberty of adding one of my favorite bootstrap references by Canty, Davison, Hinkley & Ventura (2006, CJS). If access to CJS is difficult to obtain, there was a technical report version of it on one of the authors' websites (somewhere in .ch domain, as far as I can recall).
This is a clever approach. When you "cheat," do you check that the error variances are approximately the same in each model? And if they appear substantially different, how does that affect your recommendations?
What about the distribution of the values? Are they supposed to be Gaussian?
Further, your reasoning is unsound: consider estimating the mean parameter, $\\mu$ in an exponential $f_X(X)=1/\\mu\\, \\exp(-x/\\mu)$. The distribution is skew, yet the MLE (the sample mean) is unbiased in that case. On the other hand, consider the MLE of $\\theta$ for a uniform on $(-\\theta,\\theta)$; the r.v. is symmetric yet ML is biased. Your above reasoning says it will be biased in the first case (and, presumably, not in the second). [However, this isn't the central issue; bias or no, I worry about why you ruled out ML estimation on the basis of bias alone.]
"You should know how to use multiple tools if you want to build anything interesting." Not necessary if one tool is Turing complete. But I have seen that different programming languages are useful for different problems.
In many cases (e.g. the usual $z$ or $t$-tests) the distinction literally doesn't matter since the probability that the $p$-value is exactly .05 is 0. This is the case any time the null distribution is continuous.
I understand now! You were most helpful. Much appreciated. I believe I'm ready for some real "Poissonic" action hehe.
@Neelam no problems. PCA = Principal Components Analysis, a data reduction technique: see this excellent tutorial by Lindsay I Smith: http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf
Variables such as actor_id are not numeric in the sense of quantifying anything that is important to your analysis.  I suppose they quantify the order in which the actors got listed, but that's all.  So you should conceptualize each variable like that as a nominal (a.k.a. categorical) variable.  <br><br> Beyond that, in order to obtain useful help, please say what type of procedure/analysis you are using and in what software.  Next, please tell what you have tried and what exactly is getting you stuck.
For any distribution there will be rejection procedures that come as close to 100% as you like--in theory. The issue is whether they are *fast and simple.* That comes down to efficiently generating realizations from a distribution having a similarly shaped PDF. (The 69% rate originates from the binary nature of the machines we use.) BTW, please take a look at [our policy](http://stats.stackexchange.com/help/on-topic) concerning [tag:self-study] questions.
Note that Bland-Altman is a widely accepted name within medical statistics (biostatistics), but not necessarily outside. The name is fitting in so far as Martin Bland and Douglas Altman wrote several very widely read expositions of method comparison, stressing this plot as one of the things to use. But plotting difference versus mean is a much older idea, being used by Tukey from the early 1960s if not earlier and by Neyman in the 1950s, and quite likely much earlier.
Thanks Eric - did you mean an alpha range of 10^0 to 10^-6 (alpha cannot be > 1)? Is there an advantage to using a log scale?
What exactly are you asking?  What have you tried?  Generally we don't simply do people's homework for them, we try to guide them to an answer instead.
@seaneaster - thanks for adding the link.  @ben, these calculations are simple in R.  However I'm not sure what form your samples are in.  You should be able to use `sd ()` combined with one of the "apply" functions.  As for boxplots, these are simple to get basic ones with `boxplot ()`.
@Alan Be aware that we (@caracal and me) are not citing the same book. CDA is the more complete one, but Laura Thompson's textbook already includes about 10 pages of summary for each chapter. Look at her textbook first. ICDA is available on Google books I think.
thanks for the suggestion I will look into it.  My biggest concern is the comparison though.  I think for most of the curves I could get a decent fit using a variety of approaches but I don't know how to then compare among lakes.
(+1) I like how you framed the problem in terms of link prediction. Just for my own interest, what resources have you found to be most helpful in introducing this concept?
Ah fantastic! You've just saved my whole day, good sir, with all your assistance. I'm using R, so I will try -1. Thank you so much, again, for your help.
Thanks I will give this a try
I still don't follow and I suspect many other readers would not comprehend, either. Time series are not linear or nonlinear: our *models* of them are (or are not).  It's pointless to ask for a characterization of nonlinear models: they are innumerable.
Hi thanks for the feedback.  I'm actually using "number of cigarettes" as a predictor rather than an outcome. How would you guys in marketing deal with this?
Please show your code
Did you check the assumptions of the model? Are there outliers? Are there data entry errors?
Is "55,7" a typo or 55.7 written in European format?
That is an improper accuracy scoring rule so will give misleading model comparisons.
Can you give an idea of the size of the dataset (i.e. T=?,S=?)? Is there a reason why you don't model the time varying correlation matrix directly (DCC garch)?
(+1) A good companion textbook is [Clinical Prediction Models](http://www.clinicalpredictionmodels.org/), by EW Steyerberg (especially for those interested in clinical outcomes).
"Scaling" can mean a few things. (1) It can mean linear transformations of data $\\mathbf{X}$, such as $\\mathbf{X^{*}} = a + b\\mathbf{X}$, where $-\\infty < a < \\infty$ and $0 < b < \\infty$; or (2) that the individual variables in $\\mathbf{X}$ are all measured on the same scale and have closely sized variances. My comment applies to both of these meanings.
@Louise I only have experience with rJAGS unfortunately. You can try asking [here](http://sourceforge.net/p/mcmc-jags/discussion/610036/). Martyn Plummer is the one who created JAGS and he appears to be active on that forum.
thanks. It has given me the necessary thought process to move ahead.
@Peter IMHO, this is a border case: handling of NA/Null/Nodata values is of general interest in statistical analysis.  There appear to be some lessons in this example that apply beyond only `R` applications (even though at present that good stuff is found primarily in comments :-).
These are good remarks, but they seem never to address the question itself, which comes down to making a one-tailed versus a two-tailed test.
I believe you will find my edit helpful. You might not use R frequently but here you are, spreading light to my ignorance..
Question for clarification ? You have a bunch of images, you (as a human) see what the driveable region is but you don't have this information yet readable for machines ? And now you want to segment the image to detect the driveable non-driveable regions ?
@matejuh I just added few more lines about the first approach that might direct you to a solution.
$P_{max}$ does not exists :) perhaps you meant $P_{frac}$. However, the statistical error is $e_i = X_i - \\mu$, where $X_i$ is a value of the normal distribution and $\\mu$ its mean.
"A statistical error (or disturbance) is the amount by which an observation differs from its expected value, the latter being based on the whole population from which the statistical unit was chosen randomly". Hope now we are defining the problem a little bit more precisely.
Thank you for catching the typo--it was a vestige of an earlier name for the same variable, and so the code worked on my system. (Which is a relief, because it means the output was unaffected.)  The non-zero intercept reflects the overall average success rate: it has nothing to do with the presence or absence of other techniques.
Have a look at package cbmodels at https://r-forge.r-project.org/R/?group_id=366 . It is still under development and not officially released, but you are welcome to test it. I'd be grateful for comments, my email is in the DESCRIPTION file of the package.
+1 ... EM is not only for the missing values case, though.
That does not seem to be a permanent solution.
Huh. The definition I was familiar with was the one from the introduction: "Volatility is normally expressed in annualized terms, and it may either be an absolute number ($5) or a fraction of the mean (5%)." I'm not a finance guy by any means, though, so if you or someone else wants to give a more thorough answer then that would be welcome.
Do you know anything about the arrival times to $l_1$?
@Matthew, are you the person who asked this question?  If so, we need to merge your accounts so you can more easily edit and comment on your question.  Please let us know.
I hate to sound like a complete moron, but would you mind elaborating?  How does removing "Alex" and "Steve" change the likelihood of the statement occurring?
Yes, I agree on the linear structure assumption, but on the other hand, one can always linearize pieces of the data. It is hard to know for sure if this is a reasonable transformation, or if it introduces problems down the line.

My original intent was for PCA to be done on the output; think of a physical experiment where 1 dimensional stimulus is applied to 3 dimensional instruments. However, it is generally up to the experimenter to choose something sensible. Seems like input PCA would primarily reveal flaws in the experiment design (does the choice of input numbers cover the dimensions)
Great, thanks!  For future readers, once you've done the dot product to get a number (which is really the Y coordinate in linear space... the answer to a linear regression problem in terms of log-odds) you transform it back into probability space by doing the following (where p is your answer):  exp(p) / (1 + exp(p) ) .  The resulting answer should be between 0 and 1.
This reminds me of the first Figure in  [‘The Disease Detectives’: a comic perspective on outbreak investigations](http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=3315).
scaling and centering simply refers to how you treat your data before calculating a covariance matrix (i.e. as applied to the columns of your data matrix). This does not affect the decomposition itself. Scaling may be desired for some situations where you want to remove the importance of differing variances between sample etc.
I'd vote for this one.
I'm afraid I don't know the "usual discrete way" of generating these confidence intervals, nor does your $\\inf$ notation make sense to me.
I also was thinking about reconstruction accuracy, but why you call it quick-and-dirty? Also I choosing not the more precise algorithm, but I compare "ideal" canonical slow implementation with more fast implementation which use some approximations, so it's accuracy will be less.It can be more or less accurate, but how to measure "degree of accuracy" and determine if it appropriate for real tasks?(I don't know how it's called maybe numerical stable?)
Thanks for the information. I downloaded the package and it leads me to some Fortran source code! Lol now I need to understand it and reimplement in Python.
(i) Why would that count be Gaussian? One would expect it to be quite skew. (ii) why construct a symmetric (two-tailed) deviation, when one is only interested in the upper tail (a hashtag with an unusually low frequency isn't '*breaking*')?
This question appears to be off-topic because it is about finding a freeware database program.
Ack, sorry, I totally messed up the shape of my data when constructing the fake example (the real data is HIPAA-sensitive).  My true observations aren't paired, there are a different number of x and y observations in each group.  Maybe I should kill this question and start a new one.
I've used the code blocking to try to salvage the table, not perfect, but certainly easier to read than with all of those horizontal rules. See the [advanced markdown editing help](http://stats.stackexchange.com/editing-help) for tips, and welcome to the site. Feel free to edit to improve.
@Zach, +1 for this question, I wonder is there any update of how one can do parallel processing with `caret::train()` for `Windows`, most of the examples of `APM` book are computationally expensive, at least for me 3GB RAM, 2.1GHz, dual core, 32bit Win. Had I known this issue before, I would change to `Linux`, but it is too late for me now to do such a thing. Do you know any idea of how to combat this issue in windows? if the answer by `mbq` is still active, can you pls just show in code using a concrete example of any model with moderate data size of how to implement the `computeFunction`?
can you give us an idea of the size of your model? At the very least the order of magnitude of both the number of observations and the number of coefficients. You might also look at the **biglm** R package. The **lars** package does not use **lm**, though. Be aware that **glmnet** does *not* compute the full regularization path, but instead discretizes it.
Where did you learn that statistics sole all the problems ??
Is all this really necessary when the O.P. merely requests a set of *descriptive* statistics?
I'm not sure I understand the question. You want SPSS to impute any possible values EXCEPT 8 (not applicable)? You don't want SPSS to impute 8s, is that it?
First, I'd like to thank for your answer @bmciv. However, to prove that $\\hat\\sigma^2$ is optimal we have to show that $\\sqrt n(\\hat\\sigma^2-\\sigma^2)\\to N(0,2\\sigma^4)$, and it suffices to show $\\sqrt n(\\hat\\sigma^2-\\check\\sigma^2)\\to 0$ in probability because we will have $\\hat\\sigma^2$ and $\\check\\sigma^2$ have the same limiting distribution and we already had $\\sqrt n(\\check\\sigma^2-\\sigma^2)\\to N(0,2\\sigma^4)$. In your answer, we are only able to derive  $(\\hat\\sigma^2-\\check\\sigma^2)\\to 0$ in probability. So, how will we confirm the optimality of $\\hat\\sigma^2$?