tttjjj/elt_bench huggingface dataset:
@Dan Thanks for your reply. It's been a few weeks, I actually decided to use normalized price ratio for the test. I still do not know the advantages/disadvantages of the three. As for using daily returns in link #2, it does indeed say **not** to use it, I got confused as he uses the regression of daily returns of `A,B` to obtain `beta` :). In terms of results, it seems using Price ratio has worked well. I can provide more information if you need, the full context is that I needed to generate and select co-integrated pairs for Pairs Trading.
@bill_080 I think you're right, but conventionally we would say we are matching the *99%* (i.e., upper) percentile of the 52.8% distribution to the *1%* (i.e., lower) percentile of the 55% alternative.
You don't divide the data, you _weight_ the data. I added a little bit more formal explanation to make this more clear. I can't say what kind of algorithms or automatic mechanisms @MarcClaesen had in mind when he wrote his answer.
"how do I accept an answer on this website?" There are empty check marks under the posted answers to your questions. Click one to make it green. Here's a discussion on accepting answers on meta: http://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work
@Killercam Thanks, and good luck!
@Cesar: Thx. Sure it is expected, a less illustrative way to state that is the variance formula for proportions (see my answer): the more extreme the true error rate is, the lower the variance, the maximum variance is at error rate = 50%.
Also of interest: [Is R^2 useful or dangerous?](http://stats.stackexchange.com/questions/13314/is-r2-useful-or-dangerous).
(+1) great perspective. I am pretty sure the term surrogate will stick.
It's the population mean I'm referring to.

The first set is alike a normal distribution, however with discrete values, while the second shows similarities but obviously as experimental data is variable. 

The number of observations are around 2000. Normal distribution approximations would likely be fairly close to accurate with the number of observations taken, but I was hoping to be more accurate than that and stick with discrete values.

  Forgive me if I have used any terminology incorrectly, it's been a while since I have dealt with discrete random variables.
Another good reference is Fisher, N.I. 1993. Statistical Analysis of Circular Data. Cambridge: Cambridge University Press.
[Asked and already answered](http://math.stackexchange.com/q/303728/15941) on math.SE
As of PM June 18, 2012, this is the closest answer to on topic.
I think you are thinking of it in a weird way. The real question is "can the topics discovered by LDA occur in datasets with small vocabularies". Or conversely, "are small vocabulary documents likely to produce topics that LDA can handle". LDA as an algorithm certain works on small documents, but the topics/document types that occur in such small vocabularies might be very different from what LDA is normally used for.
@Faisal: In real life applications, the true standard deviation is never known and thus replaced by any consistent estimate. To account for this "extra" uncertainty, better use Student's t-test. It yields more reliable results for small to medium sample sizes.
@DouglasZare, can you elaborate more on that? (maybe an example will be good). I'm really interested in this question, but I'm not sure I really got you suggestion.
Hi Max, how is the score from the test use?. Assuming the possible total score ranges from 5 to 15, are the incremental changes in score important?
Thank you AdamO! your a star :) Just hope i get this right now!
Typo? Perhaps the years you want to forecast are 2009-2150?  That's hard enough.
Can't figure out how to combine what different types of information?
In small area estimation, you have the same problem with "out-of-sample" small areas.  What is usually done is that you estimate the out-of-sample random effects by zero (their most likely value - assuming your random effects are normally distributed).  Effectively you are using the "synthetic" or fixed part of the model only for prediction.
Hi, I have 25 controls but only 12 patients. Each of them have a spectrogram (time x freq x channels) that is collapsed down to 1 vector.I have tried PCA on just the controls and get a good 1st PC (its not dominated by any one subject) and looks like the individual data. With patients alone, I do not get this. If I do the PCA on both patients and normals together, the 1st PC still looks like the normals but it does'nt seperate the 2 groups. SO it was suggested to me to take the PCA of just normals and then project the patients into this space but how to do that is unclear.
@Tomek Good catch: I simply left out the most crucial part of the definition of $\\delta$!  It's in now.  It is the obvious one: when the argument of the square root is negative, just set it to zero.  In other words, estimate $\\alpha = 1/2$ whenever $n/4 \\le k$.  (There was no such error in the plots, however: this was only a typo.)  $\\lambda$ is simply a number but it can change from one $n$ to another.  For a fixed $n$ that makes $h$ a linear function of $k$.
I believe their are numerous ways PROC MIXED can calculate degrees of freedom for fixed effects.  Have you tried some alternatives in the model statement?  See the `DDFM=` option specifically.  I believe the SAS For Mixed Models authors often suggest the latter options (`SATTERTHWAITE` and `KENWARDROGER`).
Ah, the suffix as a whole idea is interesting. Have you thought about doing a sort of "cumulative suffix" representation? Here, each feature would be the last N characters, so you'd have something like: 
e, ue, que, ique, tique, F 
for esthetique. My gut feeling is this more closely matches how French grammar actually works.
For simply fitting a curve, you could use a non parametric smoother, say a spline or a loess model. For the rest, I am afraid I lack the substantive background to understand it (perhaps you can provide more info to that end, like what is a diffusion coefficient and what chi-square statistic are you interested in, or what is a half life for the change of intensity?). Is this something you can derive from a smoothed curve (I guess for the half life it should be possible)?
For fitting mixture models I use mixtools package in R and it works well
You example is very helpful. I had another problem in learning Gibbs Sampling. It is hard for me to implement it. Could you suggest me some introductory books or notes that provide simple examples or codes to implement Latent Dirichlet allocation using Gibbs sampling?
thanks for your reply. Excuse me, I'm not sure I understood the question... I use SAS with default settings including for mean square... Are there any advances beyond Tukye's HSD?
@PeterFlom , Thank you very much. This time I try to predict whether EURUSD up or down tomorrow. Does it make more sense to set close - open > 0 as 1 and vice versa? In my opinion, it is much better than dividing volatility empirically by selecting a threshold randomly. Up or Down is quite black or white. I will try your method to not dichotomize the target variable then report to you. Thanks again.
Nice question.  But please run `set.seed(1234)` (or a seed of your choice) before creating the sample data to make the results reproducible.  Thanks.
*Completely pedantic side note: "data" is plural (the singular is "datum"), thus "these data include" is grammatical, whereas "this data includes" is not.*
As long as you bin the response variable without considering the values of the covariables, I doubt that this validation is useful.
Precisely. Thanks!
+1 for the definition of accurate.
You must give more context. What else do you know about the data? How big is the distance matrix? You want a classification. ¿Do you know class membership in some cases? without more context not much can be said!
For perfoamnce-tuning, see also [SO: "How to improve randomForest performance"](http://stackoverflow.com/questions/23075506/how-to-improve-randomforest-performance/23714357#23714357)
thanks for you answer. I read some stuff about but still a bit confused about "folding in". You are saying that I should keep the topic assignments of the old documents the same, this means that the word-topics assignments should be re-calculated? Is it possible for you to give me a more detailed steps of what should be done? or maybe referring me to a paper or a link that can actually help me clarify this "folding in" process. My first option is to do "folding in". If unsuccessful I will go for the second method you proposed(not sure how well it works comparinfg to folding in).thanks.
I examined the issue in detail and [got the answer](http://stats.stackexchange.com/a/90492/5509)! If you just change `dunif` to `dgamma(0.01, 0.01)`, you will start to observe the problem :-) See http://stats.stackexchange.com/a/90492
I can't help with `R` because I'm not its user. But [correlation _is_ cosine](http://stats.stackexchange.com/a/22520/3277) for centered data. So, if you need cosine between data columns, do as if you are computing $r$, only do not center columns (or, equivalently, do not compute sum-of-squares of deviations, compute just sum-of-squares).
Exactly *how* does your formula differ from `cov(D)`? For instance, is there a constant ratio of the components of `cov(D)` to the components of your calculation?
+1 Good answer.  One might also observe that $\\mathbb{E}[e^{t X}]$ is, by definition, the [mgf of the Standard Normal distribution](http://en.wikipedia.org/wiki/Normal_distribution) and therefore equals $e^{t^2/2}$.  Set $t=-\\mu$, QED.
Perfect, nice and elegant trick! I added some new information in my answer, to formalize better what I'm looking for.
There is also an ordinal logit for categorical dependent variables
I have used Latex to alter the presentation of your question.  If this is not what you intended, let me know
Both HMMs and Latent Dirichlet Allocation (I add for having more extensive references) are, to my view, [generative] language models, extensively used in NLP, and in Text Classification as well. As language models, they aim to predict the next output symbol (word) in terms of the previous ones, so in fact they try to capture language order. So, in short, I believe they should do the job. LDAs are more trending nowadays.

Alternatively, you may try deep learning, as multilayer neural networks can capture hidden relations among closely located words. And they are trending in NLP as well.
Good point.  Y now defined more clearly above.
(I know I keep saying that)
I am just wondering how they managed to get both Intercept AND Estimates. Do you think that they did it the normal way and then used the Intercept from this first run and then got the estimates with your second method? 
I assume that you would then add the mean back in (I am not sure whether that would be the overall mean of Y or the mean for only when that factor level is present. Is this a presentable method or just trying to get it the way I want without reason?
However, I await with patience for your additional comments. I understand that you must be very busy.
A simple graphical way is to calculate the coefficient of variation and graph it for the 15 variables over the 5 time points. Testing for regression to the mean is meaningless (pun not intended!), it is mathematical property. Economists have a literature on "convergence" that may be a more meaningful way to look at the series over time.
Hello, I am not sure what the etiquette is about posting code and I am not sure if I am committing a fauxpas here by posting a link without further explanation. Unfortunately, the code is too long to simply post here, thus the compressed version. Basically, the implemented algorithm utilises an ensemble of neural networks to find a mapping from a 2D coordinates that live in a 2-dimensional space to the high-dimensional data.
A very good tool indeed, which also works with R.
But seeing the normal distribution shape form from the experiment is really eye-opening.  I first saw one of these demonstrated at the Science Museum in Chicago when I was a child (about 10 years old).  Didn't know anything about statistics then but never forgot the curve.
Of possible interest: [How to visualize what ANOVA does?](http://stats.stackexchange.com/q/5278/930)
Can you include a specification of the model you're trying to fit?  In a comment to Gavin's answer, you mention several covariates.  What model structure are you seeking?
Good point, the fact that it would concern a very small proportion of the population. And indeed, if ± 2 SD yields smaller p-values than ± 1 SD, why stop there...
@Glen_b, that is very true, but the point is can one, *mathematically* claim(prove) that then these distributions are assignable to new random variables each. Intuitively this make sense as all of these pdfs are different but I am not sure if I am not saying gibberish.
@Glen_b It's a mix of exactly 0 and 1's and continuous values between 0 and 1.

If they were ratio of counts, couldn't there be exactly 0 and 1's as well?
@PeterR good point! Fixed the error.
Since the population mean is known, so just subtract it from your data points, or from your one-sample CI.  It is rare for the population mean to be known so the software probably won't do this.
This s perhaps not a direct answer to the question, but rather a hint. If you are an R user, you might find it useful that the bivariate MacKay distribution is implemented in the VGAM library, using the function bivgamma.mackay. I am using it myself, and finding it very useful, although, as noted by Xi'an, it has the strong limitation that Y must be strictly greater than X.
@JohnShedletsky that's ok. If you've really got gigabytes of data to work with you can probably fit a pretty darn complicated model. In any case unfortunately you don't get to 'not assume' the form of the solution. Any model you choose assumes some form or other, even the SVD, and even so-called 'non-parametric' ones.  But if your aim is to predict well and maybe experiment later, then being misspecified might not matter at all.
Ok thanks, it is the mean we want to test as all our other output has been in this form.
I think you meant countable and not necessarily finite, am I wrong? Or am I mixing with integrals.
Yes I am going to use Cox regression for that most likely. But the risk factors identified will reflect which period of time? I think it will reflect the median time in the data (40 days)? Because I want to answer a question which is predctors of healing (or non-healing) within 2 months of discharge.
What makes you say "Cramer's V says there isn't a significant difference"??
If you tell us a little bit about what you've tried so far and where you got stuck, we may be able to give you better help.
Hi, It's just an assignment for school here. I also need to argue why p-values are a bad idea, so you helped me out here! Thanks
In particular, noting that $p_A+p_B=1$, your last expression simplifies to $\\sigma^2=\\mu^{(2)}-\\mu^2=p_A\\sigma_A^2+p_B\\sigma_B^2+p_Ap_B(\\mu_A-\\mu_B)^2$.
@Vara I read in the meta for stats that R questions are appropriate here. There may be more R users on stackoverflow.com, but I would expect more R users with knowledge of panel packages on stats.stackoverflow.com.
"compare A and B" isn't really the same as "see if there's a relationship between A and B". It sounds like you need something like regression, whereas your title suggests something else.
I thought that the rank of this matrix has two solutions. If $T=K$ then the $tr(M)=0$ otherwise the $rank(M)=N$...
I clearly lack the type/level of intuition needed to understand what's meant by a weak tendency of nature that doesn't exist in a measurable way. I'll stick to my statistical last & leave the more highly-evolved souls to contemplate it in peace.
1.The estimator was not construct manually, I use density function  provide as default in R. 2.the problem is related to density decomposition, there is no assumption about the "noise". 3. condition $X \\sim Exp(\\lambda)$ was mentioned above. And $Y$ as I guess (through the result with negative value), it might be mixture of two normal distribution.@julienstirnemann
I am working on events of 4 spins
I'm sorry for being lazy to read all your question, but... when there is a categorical DV repeatedly measured and there is also between-subject factors and covariates, the first thing what comes to mind regarding SPSS is Generalized Estimating Equations procedure.
Is this a homework problem?  If so, you should tag it as self study, and lay out to the readers of CrossValidated where you're stuck and/or what you've tried/considered to this point to try and solve this problem.  thanks.
I thought this was a very well stated question. Clear and concise. +1 for that!
If you were will to start simply (walk or crawl before running) you might wish to try http://stats.stackexchange.com/questions/41794/bayesian-updating-for-a-discrete-rating-value/43048#43048 with other intro here http://stats.stackexchange.com/questions/44244/ap-statistics-exam/44265#44265 If you wish, I could provide some initial R code.
The title question and the one in the body of the post don't seem to quite correspond. Which did you want answered?
Welcome to the site, @user27393. Are you asking about this from a general / conceptual / theoretical perspective, or just about how to get R / Rpart to give these to you? If the former, the question belongs here, but if the latter, it would be off-topic for CV (see our [help page](http://stats.stackexchange.com/help)), but on-topic on [Stack Overflow](http://stackoverflow.com/). Please edit to clarify your question, if it is the latter, flag your Q & we'll migrate it for you (*please don't cross-post, though*).
You really need to say more about the context here. @ChrisA. has made a notable attempt, but it's hard to know if your question is really answered, because it's difficult to know much about your quandry.
Hey Alex, when you talk about instrumental variables and you say "affect the independent variable of interest", does affect mean 'correlated', or does affect mean you know that this instrumental variable has a casual affect on your interested ind. variable
What is a ROC curve? Could you please provide a link?
your explanation is nice. It is followed by valuable supplement which caused an interesting series of comments. Thanks a lot to all :) !
This question appears to be off-topic because it is about errors in software. If you add a minimal reproducible example and follow the other requirements described [here](http://stackoverflow.com/help/on-topic) it might be on topic on StackOverflow.
Can you provide your  code?
As a non poker player, does order matter. For example, would 3-6-4-5 be a different hand? This influences the answer. What about suit of the 4 cards?
(1) I don't use JMP so this guess (2) the model shown looks like it is for standard dummy coding and the coefficient estimates shown above for the 'reverse helmert' coding (3) Regarding question: No. if the difference between dummy coefficients (2) and (3) are not significant this had nothing to do with a comparison of (3) and (1).
Classical MDS takes as it input the pairwise Euclidean distances among n objects and returns the coordinates of n corresponding points in an m-dimensional space whose origin is at the centroid of the configuration, with the reference axes rotated to the principal axes of the configuration. Given the input distances of an (n+1)st object to the n others, the general principal to be followed in augmenting the solution without rerunning the analysis is to place the corresponding point in the m-space so that its distances to the other points approximate the input distances in some sense.
@Aniko : I agree, malformed is a wrong word, but I didn't really know how to say it different.
The output from PROC GLM and PROC REG includes the number of observations, it's the first thing in the output. It is also in the log file. I am not sure what you mean by "adding it to the dataset", the number of observations is not part of the data.
The best way is to use substantive knowledge and ignore statistical significance during model fitting. If you must use some automatic method, LASSO or LAR are good. See my paper [stopping stepwise](http://www.nesug.org/proceedings/nesug07/sa/sa07.pdf) for more; or see @FrankHarrell 's book "Regression Modelling Strategies".
"Minka says." :)
I have not checked the details of your calculation but an obvious error is that the purported marginal density of $X$ shows dependence on $\\theta$. You need to apply the limits $\\theta=0$ and $\\theta=1$ to the $\\thetae^{-\\theta x}/(-x)$ also...
Homework question should be tagged as such, because they receive a [special treatment](http://meta.stackexchange.com/questions/10811/how-to-ask-and-answer-homework-questions/10812#10812). Users who are always ready to assist you probably expect you to show what you tried to solve those questions.
Your first two questions are answered in the duplicate thread at http://stats.stackexchange.com/questions/35711/box-cox-like-transformation-for-independent-variables, so please focus on the third question here.
@Srikant. No! This is how classical CIs bite. Let's assume for simplicity that the amount of cereal filled in a box is normal with mean $\\mu$ and variance $\\sigma^2$. The confidence interval of $\\mu$ is based on its *sampling* distribution which is different. A particular CI may be way off due to sampling errors and then it will have no relation to how the machine performs. If you were to repeatedly sample and repeatedly form CIs then 95% of them would be right, but that is no consolation.
Incidentally, the unavoidable appearance of $S(d,k)$ in the solution (it does not simplify further) indicates that although the Markov chain methods will work, it will be moderately difficult to carry them through in general and will not lead to nice closed formulas: it will be tantamount to rediscovering the Stirling numbers.
Yes, that is correct. It is nearly identical to how you interpret positive coefficients.
Thanks for the detailed answer.  And yes, the reason I'm doing this running the full model (with no down-sampling) is computationally prohibitive.
Please explain *how* you want them to be dependent--there are many ways this can happen, after all.
I am not sure I understand the question. What do you mean by "margin error support vectors"?
can you not still use poisson but with a log population offset term so you are modelling the rate?
I have seen *normalized* used to suggest *standardized* or to suggest fitted onto a standard normal distribution i.e. $\\Phi^{-1}(F(X))$, so of the three *normalized* is most likely to be misunderstood.  Ada's comment of the application of a *[normalizing constant](https://en.wikipedia.org/wiki/Normalizing_constant)* to a likelihood function is yet another possible interpretation.
You do not necessarily increase accuracy by increasing number of bootstrap replicates. In fact, if you run too many trees you run the risk of over correlating the ensemble and causing an overfit problem. There is also a balance between error convergence and accounting for interactions, which stabilize at a slower rate than error. Plot your RF object to assess error convergence. As a rule of thumb I run twice the number of trees than required for error stabilization. More parameters require a longer burn-in.
"*The CLT says that sum of many random variables will be normally distributed even if their underlying distributions are not normally distributed.*" --- well, not quite. It's not actually a theorem (or rather a collection of theorems) about sums, but about standardized averages (such as $\\frac{\\bar x-\\mu}{\\sigma/\\sqrt n}$), and it doesn't apply to every possible distribution.
+1 for the answer and the R code. looks like i made a mistake!
Thanks! I tried it without and with seasonal.period=365 but it doesn't seem to capture the seasonality well. Also, longer forecast horizons seem to be a problem. Here are the forecast plots for TBATS, ETS, and HoltWinters (that seems to perform best right now).
http://timalthoff.de/data/bats.jpg
http://timalthoff.de/data/ets.jpg
http://timalthoff.de/data/HoltWinters.jpg