tttjjj/elt_bench huggingface dataset:
No problem, happens to all of us once in awhile!
Actually, _all_ examples are periodic ($f(t+T)=f(t)$). Periodicity has nothing to do with a function being jagged or not. Sines and cosines are not the only periodic functions! To distinguish between your functions you can try $\\sum_t (f(t+1)-f(t))^2 / \\langle (f-\\langle f \\rangle)^2 \\rangle$ (it is normalized by the variance). For your particular examples it is 61.5, 62.0 and 1606.1.
IMO you should ask the second part (confidence) as a different question -- it is a broad topic and very classifier-dependent.
I think, based on your description, I would like the latter: a formula for the linear combination $\\mathbf{X}^T \\beta$ of variables that gives the corresponding PC projection score $f_{1, j}$: this can be interpreted as a latent factor score, right? I don't know where I can identify the $\\beta$ in the results.
Surprising ...Wiki has separate definitions for weak and second order but there is no reference for second order stationary.
Run both commands and you'll find out.  You're not going to break anything trying stuff out.
I assume this is homework then? This process exhibits mean-reversion, which you can model with an AR(p) model in (log) levels. Since it is not a random walk, you could not say the market is efficient. It is possible that the market is over-reacting to news and then correcting.
All in all, clustering mixed-type data is a tricky thing and might be for an experienced data analyst only, perhaps. On the other hand, clustering of such data is often not a good idea at all, because there are issues of standardization, interpretation and features contribution analysis.
Thanks, I have not ever worked with python. I will try to find an R solution
@crucified, no closed form will exist for the $t$ quantile function. See the G. W. Hill references listed [here](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/TDist.html). You could also get the source code for [GSL](http://www.gnu.org/software/gsl/) and look up the [appropriate functions](http://www.gnu.org/software/gsl/manual/html_node/The-t_002ddistribution.html). I haven't looked myself, but it's probably easy enough to port to MATLAB.
Could you specify what your question is? Is this about the relevance or validity of using repeated-measures ANOVA with appropriate test for the variance-covariance structure, alternative ways for analyzing RCBD, or so?
@ConjugatePrior Oh well I see. So generally, is the $x$ notation which is used for data-points and $z$ for the hidden variables (like in the picture), or is it the opposite (like in my equations) ?
Do you have any measure/estimate of the consequences of being out by a given amount in your forecast?
Having tried to implement a Levene test today, I've realized that I don't think it's what I'm looking for.  It extends the F test by allowing a comparison of equality among several samples.  If the F test allows a comparison of the variance of A vs B, the Levene test allows a comparison of A vs B vs C vs D etc.   However, what I'm after is a test of ((A1 vs B1) and (A2 vs B2) and (A3 vs B3)) etc. to see whether the As have difference variance to the Bs.  I've been googling for this and haven't been able to turn anything up.  Does anyone have any ideas of how I might achieve this joint test?
This book came highly recommended to me, and we now have it in our library at work: http://www.amazon.com/Applied-Longitudinal-Data-Analysis-Occurrence/dp/0195152964/ref=pd_sim_b_1
Hi Erik and thanks. You know, I almost posted the question on mapleprimes, not SE ! I ran your code and also got good results, albeit with a "Warning, undefined value encountered"
As stated, this question is too broad to be answerable. Can you clarify "cost and error" to make your Q more specific?
Thank you very much for the reply. Our hypothesis is that fold change would be same across all chromosomes, but smaller chromosomes seems to be more active for event-Y. So it will be nicer to show difference in terms of significance level. Thanks again for the suggestion, I will check out rms (I am an R user).
Sounds like Simpson's paradox, to me.  Wikipedia has a nice explanation of this
@Glen_b is right. [John Fox](http://goo.gl/jtroHD) writes on pages 35-36: "The standard error of the order statistic $X_{(i)}$ is $$\\mathrm{SE}(X_{(i)})=\\frac{\\hat{\\sigma}}{p(z_i)}\\sqrt{\\frac{P_i(1-P_i)}{n}}$$ where $p(z)$ is the probability density function corresponding to the CDF $P(z)$. The values along the fitted line are given by $\\widehat{X}_{(i)}=\\hat{\\mu}+\\hat{\\sigma}z_{i}$. An approximate 95% confidence "envelope" around the fitted line is, therefore, $\\widehat{X}_{(i)}\\pm 2\\times \\mathrm{SE}(X_{(i)})$."
I guess it depends on which test statistic you mean. For example, the z-test for an autocorrelation will likely be almost identical to the LR test (though I haven't worked out the LR test, I think from the form of the likelihood for an AR(p) (which apart from a term involving the first p observations, looks the same as a regression) I think it should work out.
The people at Google run logistic regressions with hundreds of thousands of predictors (mostly dummies) to decide what ads to show. (Obviously they have some fairly large samples).
The converse problem with frequentist statstics is that the subjectivity is there, but it isn't mentioned at all.  The (practical) trouble with likelihood ratios is that they are based on optimising the likelihood and hence ignore the fact that there may be other solutions with a likelihood only slightly less.  That is where the Bayes factor is useful.  But it is always "horses for courses".
Could you explain how MCA would help here? As I understand it, MCA usually deals with a set of categorical (most often nominal) variables, and doesn't consider blocks of variables, in its basic formulation. Moreover, it doesn't incorporate a model for measurement errors (attached to each item/variable), which FA does, for instance.
@whuber Thanks for your comment. It does indeed look like this method produces estimates regressed towards a mean and that's not what I want. Can you give me any pointers on reading about using simulations to make realistic predictions? I have fitted deterministic trigonometric models to the water temperature data previously but I would like to use the air temperature data to help fill the gaps.
I would advise you check none of your variables are constants (ie no variance). If you're fine on that score check whether you have any complex valued variables or infinite variables.
@fbielejec: Ok, thanks for the definition. You random variable is definitely discrete, thus it cannot have "density equal to P{X=k+1} for x∈[k,k+1)" since your variable cannot have, for example, value of 1/2. The density is zero between integer points, all you have is just a bunch of atoms.

Thus, the cdf is (as usual) just sum(pebinom(0:floor(x), N, p)) (note that you broken the naming conventions, your pebinom should really be debinom, it's a density wrt counting measure, and debinom should be pebinom).
@Glen_b - LOL! You could've done it too :) Updated.
I know it is late, hope this helps you or anyone else ... Calculate the log likelihood for each sequence ..
Sorry. That was a silly mistake. Thanks for pointing it out.
Take a look at [the Categorical distribution](http://en.wikipedia.org/wiki/Categorical_distribution).
According to Table 16 of this article...  http://www.cdc.gov/nchs/data/nvsr/nvsr58/nvsr58_24.pdf... the expected number of births for April should be a little less than 12.  Typically peak births are late September with the max day one gestation period following New Years Eve... so get ready for Saturday!!! woo hoo!
+1, I enjoy Borsboom as well. For those interested in The Attack article I think would also be interested in "The Concept of Validity", http://rhowell.ba.ttu.edu/borsboomValidity2004.pdf . Although it is a little more verbose so it is not as easy to follow as the Attack article.
A minute for 1,000 variates isn't very good: you will wait hours for one good Monte-Carlo simulation.  You can go four orders of magnitude faster using rejection sampling.  The trick is to reject with a close approximation of $f$ rather than with respect to a uniform distribution.  Concerning the calculation: compute $a \\log(c d) - \\log(\\Gamma(a))$ (by computing log Gamma directly, of course), then exponentiate.  That avoids overflow.
Maybe you could find this paper useful:  http://www.ncbi.nlm.nih.gov/pubmed/22253321
are there any sas related good blogs as well ?
Who said that having 120 cases for 50 cases is bad? I don't see why it would cause any spurious associations. However, there might be other issues (not the N, per se) such as the controls being different than the cases in some way.
See [this paper](http://www.sfs.uni-tuebingen.de/~hbaayen/publications/baayenDavidsonBates.pdf) by Baayen (2008) for a great discussion on this issue. Just as a side note, I've found that inverse transformation typically work best to fix the skew of the residuals, and it can be interpreted as items/second, or speed.
(-1) I believe this post shows some fundamental misunderstandings on several points. It's hard to know even where to start.
By the way if you are using random forest you don't need to recode your features into dummies. Just use them as is.
@Tommyr I will leave the answer to your homework for you to choose. Your professor and I may...disagree on some things.
The number of effective parameters is not the same as the sample size.  The variance of the empirical cumulative distribution function is about the same as the variance of a parametric fit to the distribution when the distribution has 4 unknown parameters to estimate.  One reason is that the empirical CDF estimates are forced to be in ascending order.
I've been doing a lot of research into clustering algorithms, and I finally know enough now to fully appreciate this reply. Thank you again, Anony-Mousse.
Not the answer to the question - but still a good answer:  +1
The line you see is not fitted. The data gives you a similar graph like michaellis menten, but in that model there is no time involved, here I have response ~ time not velocity ~ [substrate concentration]
Thanks, it is clearer now (:
I've corrected the typos I found when re-doing the computations to get the final expression. Feel free to do the computations yourself to double check.
Thanks for you answer. I am a complete amateur statistician and I got it because of you. I use EViews, it is the p-value which is 0, not a problem though I just have to interpret it. I followed some papers on the same topic when I set the IV, obviously it is hard to select instruments. Thank you again, much appreciated help.
@Gunnhild, your measurements are unlikely to be correlated, but likely to have the same bias. The bias may be correlated with the surveyor,
Ron, I have added your change to your question and responded to it. Please note that the way you worded it in your comment doesn't make sense (what's the "fifth quantile"? What's the "first quantile"?), but with that part left out, we can get somewhere.
@probabilityislogic would you please explain a little more
Pedantic note: Bray–Curtis is not a distance but a dissimilarity.
@TalGalili: edited based on your comment; also, consider putting this in the question, see my comment to the question for details.
That's fine--I agree you seem to have transcribed the formula exactly correctly--but unfortunately as it stands in that reference, the formula is nonsense. (Do you see how the product of $\\hat\\lambda$ with $du$ introduces *products* of two differentials in the integrals, in the form $dN_{1i}du$ and $dN_{2i}du$?  And the outer sum seems to have been introduced as a discrete approximation to the integral, but there's no explanation.) You will need to back up a page or two and reproduce the calculations yourself and perhaps even guess what was intended. Maybe you could contact the author.
The Wilcoxon test does not strictly make that assumption if you are careful in computing $P$-values.  If you don't use the normal approximation but use a permutation exact test or use the full $U$-statistic method you can be OK.  For the latter you can run the following in the R `Hmisc` package: `rcorr.cens(x, Surv(y))` and test $D_{xy}=0$ by getting the estimate for $D$ and its standard error.  You can also use the generalization of the Wilcoxon test - the proportional odds ordinal logistic model.  See the `orm` function in the `rms` package.
I have tried using different learning rates and very small convergence threshold 10^-6. But still, it converges quickly giving me parameter values that are different from the actual ones
I hate this quote. It makes professions using statistics look like you could cheat. But, when someone profoundly uses statistics one knows that actually you cannot cheat. Because when provided with enough information about the statistical procedures used, one can draw a conclusion on the soundness of the procedures/results. If not enough information on the statistical (and other) procedures are provided, you should immediately question the results.
@Glen_b I asked a similar question here: [link](https://stats.stackexchange.com/questions/100566/deterministic-sampling-for-multivariate-discrete-distributions) I agree MC scales wrt dimension but I need to calculate such expectations a lot and to get accurate enough approximations I will need particles that scales with dimension. I am looking for something that is like unscented transform, i.e. deterministic sampling.
A better method is to use condition indices to diagnose collinearity, or correlation matrix to see which variables are correlated. I am not sure what PCA does for you here.
Thanks for the paper reference, I'll check it out. Without having read the paper though, let me restate that I too am going to use benchmark functions (the set I mentioned above) .. I just want to be able to somehow quantify that my (hopefully) better numbers are not just due to "luck" but statistically significant. I.e., I have seen people report just numbers (such as time to execute or number of evaluations), but I am not sure that's sufficient. Be curious about other comments. Thanks.
@StéphaneLaurent Thanks for that. I confess that up to this point I have not seen population r-square as a property of contention. For example, I could propose a data generating process and there would be an r-square that is approached as my simulation sample size approaches infinity. And likewise I assume that there is a data generating process for my data, and therefore if it were possible to get an infinite sample, I could calculate the true population r-square.
Hrm... This could do with a more detailed example. I have trouble seeing how this would help deal with the complexity I'm talking about. In particular, it doesn't help deal with what to do with analyses (derived data, plots, etc.) from investigation paths that lead to dead ends.
why do you want to use them as continuous variables? They seem like factors (in R) to me. If using R, just fit a `glm` including the variables as is, and R will do the right thing.
@Tomas Ok, it is noted. However, I would have appreciated a comment directly related to my question and not to the semantic.
@cardinal, @Chris. oh, I as usually missed the part of the task ^__^ In the case of binary-valued process it seems to be a very well known (studied) problem of traffic measurement that comes from communication networks or so called ON/OFF process that exhibits long range dependence (long memory) property. As for the particular example, I'm a bit confused, since in "one way to predict" Chris actually takes the previous value, not using the ACF only (or I'm even more confused by the term "classification rate").
"Nearest neighbor" is not the only variable kernel. The papers I mention use other tool such as Lepskii's algorithm. I'll read the AOS paper but as the performences of nearest neighbor should decrease with the dimension, I found it strange that increasing the dimension gives advantages to a "very non-parametric" estimator (If we admit constant bandwidth is less non parametric than varying bandwith). In this type of situation, the evaluation case that is used often determine the results ...
That correct. And that's the crux of the problem. Ultimately this question comes down to a theorem of linear algebra: when $A=U^\\prime U,$ $Ab=0$ if and only if $Ub=0$.
Regarding your answer: Conditional on the first success, do you not want the number of failures (purchases) *before* the first success (no purchase)? That is the geometric on the set $\\{0,1,\\ldots\\}$ not $\\{1,2,\\ldots\\}$ and so the expected value of this r.v. would be $\\mu = 0.9/0.1 = 9$, yielding $\\mathbb E X = 1$ as @Isaac has calculated in his question.
Well, I was thinking of things like: peak detection, structural changes, distortion in the frequency domain, etc. Giving more context might help to get precise and helpful responses.
http://cran.r-project.org/web/packages/EpiModel/index.html
I've done an on-line course with Adnrew Ng and read a few papers regarding my problem space. The problem I'm trying to address is the ability to authenticate users based on their keystroke timing patterns during login. Therefore I cant get testing data of imposters trying hack into the account for security reasons. Would it possible to generate data for the testing purposes?
@JeffG. you'll get a better answer if you explain exactly what you're doing, including at a minimum what the "other regression technique" is.
Intraclass correlation is one measure of agreement and another popular measure is Kappa.  But although rater agreement could be partially credited to the algorithm, I think it is partially and perhaps primarily a matter of individual similarities and differences and you have no way to separate these effects unless you also use these same raters with a different algorithm for comparison.
@Jeromy I see what you mean.  I was responding to your comment in answer to Alex's question.  Of greater interest to me is the idea that Excel can be used as a prototype for commands in R, Mathematica, (and even APL if anyone remembers it), provided you use its array-oriented procedures.  Excel itself is pretty bad as a statistical tool but good for rapid prototyping because it's easy to see and correct one's mistakes.
For any even moderate sized graph, an edge-vertex graph visualization is simply going to tell you much more about the layout algorithm than anything about the data itself.
Stat, so that this answer can stand by itself (and be resistant to link rot), would you mind giving a summary of the solution?
This appears to be fully explained in answers to [what is a complete list of the usual assumptions for linear regression](http://stats.stackexchange.com/a/16460).
For one approach, see the [Hotelling T test](http://en.wikipedia.org/wiki/Hotelling%27s_T-squared_distribution)
Where is "time series" in his question?
Also think about how to interpret the pvalue if the null hypothesis is false (eg the number of hairs on both sides of  sheep are exactly equal). There is nothing stopping you from calculating a p value, but the procedure of accepting/rejecting this type of hypothesis does not seem very meaningful.
It's a reasonable guess, & I may be wrong, but it sounds like you want to know 'how do I code up X?', where X just happens to be a data visualization rather than something else (eg, a hack for screwy date values). I would say that if the question is *about* the data visualization (eg, what are the pros & cons of this visualization), then it's best here, but that if it's about the *coding*, it's best on SO.
It seems to me all three of your assertions are wrong: Excel's default plots are ugly, R is well-known to produce nice graphics (and so-called spaghetti plots are easily handled within R, as @mbq said), Gnuplot has strictly no problem with multiple columns input data. But the real question is about *automation* although you didn't specify what you're expecting: Are you seeking for a script that might be launched in a given directory with a lot of `csv` files, do you want to build a minimalist GUI, should this be run locally or on a website, etc.?
And not just for the intercept, I mean...
@gu11aume.  Do you know Tom Ferguson?  He was a UCLA professor way back in the late 1970s when I was a graduate student.  At Stanford would used his book in our graduate math stat course.  It was a really good text and I think he writes very well.
The only way I can understand what you wrote is that if someone has too much time on their hands and wants to just pass the time of day, this is better than street crime.
I guess it does not have a name. thanks.
Wooldridge makes the (strong) assumption that the seasonal patterns are constant across time. So if there was systematic error in December, that coefficient should pick that up.
at least in the specific case where $x_i$ have a Bernoulli distribution I can estimate the variance of $x$ by $\\bar{x}(1-\\bar{x})$ as noted above. Even in this case, as noted in the question, I need a larger sample size than I would have expected.
Maybe you’ll enjoy a look on the so-called [Principle of Maximum Entropy](http://en.wikipedia.org/wiki/Principle_of_maximum_entropy). I don’t feel like expanding that in a full answer – the Wikipedia article seems of good quality. I’m pretty confident some contributors will expand on it that much better than I would.
@ttnphns I want to learn best practices in this area, so I will not need to invent wheel again.
What astronomical object are you observing? Stars? Galaxies? This can really help, because the SDSS team gets sub-pixel accuracy by first identifying the eigenbasis of the target galaxy via Principal Component Analysis. Check this draft by Aihara et al. (2011): http://icc.ub.edu/~liciaverde/dr8.pdf.
Please clarify the question. In particular, these were some comments from before: By @chl - "you should consider writing clear questions (they are interesting on their own), with one definitive issue, and reserve comments for additional infos relevant to your original question, not follow-up". By @shane - "Regarding this current question: it could also be improved because it's asked many different questions without a clear common thread. Are you interested in multicollinearity in general? Or are you interested in VIF? It would be better to break these out for clarity."
This probably belongs on stackoverflow
@Dmitrij, all the things you mention fall into the same category, something else is causing the relationship. That is why I cited the wikipedia.
The use of matching followed by a paired t-test is fine; there's some point to that (to control for other variables). The matching makes it paired. However, the way it was done -- the use of *the p-value of the test itself* as the matching criterion -- renders the resulting p-value meaningless. It's certainly no longer uniformly distributed under the null. The hypothesis test is effectively useless.
How does this differ from a sample space in probability theory? Just asking. I would like to know.
@whuber,  all the values of **a** (or **c**) associated with each pair _(xi,yi)_ are the same.
Is that mean there is a away to carry the analysis? can you recommend me a source or something?
@obfuscation Does [this](http://en.wikipedia.org/wiki/Friedman_test#Post_hoc_analysis) or [that](http://www.r-statistics.com/2010/02/post-hoc-analysis-for-friedmans-test-r-code/) help any?
Thanks, Enrico, for the full model.
Thank you, this may be useful.
Look into second order (or higher order) factor analysis. I think this is what you are looking for.
Hi there, could you let us know how many p-values you're talking about and the research question that has led to this technique?
@John +1 for a good answer.  I think that a one-sided McNemar test would be of interest since as was shown for humans There may be slightly more male births than female births. Also, the actual difference in the proportions may be small.  So large samp;es would be need to have good power to reject the null hypothesis.  In the example given in a post on CV the estimate was around 0.505 for males and 0.495 for females.