tttjjj/elt_bench huggingface dataset:
The Central Limit Theorem implies that the limiting distribution of the sample mean is normal regardless of the distribution of the original data (under some conditions). In many cases a sample size $n>400$ is large enough so that the normal approximation is quite accurate but the accuracy does depend on the parent distribution-- it might help if you, for example, posted a histogram of the original data.
Thanks a lot. I found https://files.nyu.edu/mrg217/public/mle_introduction1.pdf shortly after I asked the question and it essentially says the same thing as the notes that you shared say.
Have you been able to figure it out?
Welcome to cross validated! As your question is on the proper site now, have a look at the listed "Related" questions. Do they answer your question? You may also want to check the warnings about model comparison on the basis of ROC that have been written here many times.
@DouglasZare the main question is that: "How would you modify the error function and the weight update rule of backprogation algorithm" and maybe the key point is relies on that consideration: "It states that the function is symmetric around x=3.5"
You don't need *any* throws to "have 99% confidence" that all proportions are between 0 and 100%! What is missing from your question is a specification of the *precision* you want to have confidence in; without that, the specific question is unanswerable (although the general question can be addressed).
What clustering algorithm are you using?
Thanks for your answer. I guess I am looking for stratified sampling. (I was looking for algorithms, which are not computationally very expensive, as not parsing the entire population, to make a representative set, does not even make sense. :-))
@gung Yes I did, thank you: change made.
@kwanti If you have outliers then using the squared values typically causes the fit to adjust more to the outliers than if you used the absolute value of the residuals.  So minimizing the sum of the absolute residuals could be seen as more resistant to outliers.
I was thinking of this paper: http://www.niculescu-mizil.org/papers/comparison.tr.pdf  (+/-problematic) But google will produce more: http://lowrank.net/nikos/pubs/empirical.pdf    I'm not convinced by these, hard thing to study, but have found that most practitioners will push for RF or boosting algorithms.
I guess LLE means local liner embedding, but generally it's a good idea to decipher all abbreviations. Also, it would be helpful to give explicit links or citations if you mention a paper ("Karbauskaite et al").
You will need to define some score function to optimize. Do you have ideas?
You tell me! Seriously, that is a substantive question, not a statistical one. The absence rate, as far as I can tell, is simple (and that is what you asked for). It is possible that the absence rate **depends** on the number of flights, and you could include that in some models if you are looking to explain the absence rate. At the initial stage I suggested, you could certainly add "number of flights" to the plot.
@Berkan I haven't read the paper yet (thanks though!), but the reason I didn't unify the sample sizes is because 20/100 is not the same prior as 10/50, and that fact is very important to the reason this question was asked.
Found another one as well perhaps could be useful http://quantpsy.org/chisq/chisq.htm
@Developer: While I appreciate the accept, I think it is a bit premature.  Maybe you can unaccept, to encourage others to post answers?  I'd really like to know some of the answers from others, too.  :)
Thank you for the idea of indexing firm size (I use either market capitalization or book value of assets), which indeed mitigates some of the multicollinearity. Your explanation of continuous*continuous interaction terms was also quite helpful. This solution is definitely a bit more tricky to interpret, but I'll give it a try. Cheers!
+1, I think the proportional odds model (or something similar like ordered probit) is the default for problems like this.
@mmacx After reconsidering I'm not yet happy with this answer, it only calculates the probability of the two ships being simultaneously within 5 miles of their midpoint. Because the positions are being modelled with Gaussians, their extent is infinite. So there are many rare configurations where both ships may be far from their nominal positions yet within 5 miles of each other.
I guess the question is the following: "How can one simulate data with binary response and features with predictive power", i.e. an artificial dataset for a binary classification problem. @ZKHAN can you clarify ?
The CLT often fails (that is, does not even apply) in this circumstance unless the $p_i$ remain away from $0$ and $1$.
Thanks for the answer. Your Gaussian process formulation seems interesting, but as you say, it does not seem so easy to compute. The quadratic programming approach does not seem to extend easily in the presence of other sources of noise, unless I'm mistaken.
I agree with @JeremyMiles it is generally fine to think of these terms as referring to virtually the same model. But I probably should add that some people use these different terms to refer to specific classes of models. When Raudenbush & Bryk discuss "HLMs", they are explicit that they mean mixed models where the random factors are nested, NOT crossed. As for "random effects models," some use this term to refer specifically to mixed models where the only fixed effect is the overall constant term, however, economists often use the term more liberally as essentially a synonym for mixed models.
Information criteria, like the AIC & BIC, simply adjust the deviance of the model for its complexity (ie, the number of parameters). If you have the same number of covariates (not even necessarily the same covariates themselves), that adjustment will be irrelevant. You can check them by comparing the deviances directly. You may find it helpful to read my answer here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909), which touches on this issue.
@mpiktas.means can I write $$E[cos(\\pi X_{n+1})]=cos(\\pi E(X_{n+1}))?$$
@Memming, you may consider an entire daily data of an asset (say 20 years). The data is highly correlated and up/down trend prediction. Thanks!
Thanks for adding the explanation. +1.
This is an interesting perspective!
@Kazimierz, post this as an update to your original question. This is not a forum that people read top to bottom, or whatever. Additional important information should be in your main post. As a side note, the idea to set residual variances to zeroes is ridiculous. Do you really believe that there is no regression error in your latent variable model? I don't. If I were a reviewer looking at the paper that did this, I would simply stop at that place and write a single line rejection (or major revision) note saying, `The authors should relax their error variances.`
*That* (negative $\\alpha$'s) is certainly possible
Yes, it appears to be overfitting(which Random Forest Regression normally doesn't, hence the question). Now I have observed that changing the parameters has little effect with RF Regressors. Now cross validation requires an underlying model flexible enough to be optimized. Which kind of ML models/algorithm do you recommend for this kind of data.
All this makes sense to me: does it makes sense to you? I'm really open to second opinions :-)
$Y$ is normal with mean $\\nu$ if and only if $Y+a$ is normal with mean $\\nu+a$. The ratio of normal seems to be treated here http://en.wikipedia.org/wiki/Ratio_distribution (google ratio distribution ...)
But given the available information I won't be able to determine if the enchanced model is better. The question does state that I must reach a conclusion with the available information. @Alecos
Maybe so! Fortunately for freshmen, @Glen_b has offered [an excellent breakdown here](http://stats.stackexchange.com/a/88067/32036).
Nope, plenty of details in the paper. I've updated my answer with another approach though :)
(1)There's a distinction between "testing for normality" and "visually assessing normality". Those two things are different, and useful for different things. (2) neither can tell you that the underlying variable is normal. (3) You can visually assess normality by looking at a histogram, with caveats as the answers have noted. (4) You can also *test* normality on the basis of the histogram in several different ways, including using a chi-square - but beware, the chi-square lacks power and there are better alternatives that don't ignore cell-ordering; *smooth tests* are one possibility.
You need to tell us more. You may find [this post](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) from my blog helpful.
I guess it all boils down to the number of classes involved in the dataset for validating this approach, in particular to the case of LDA. Higher the number of classes, I guess, less performing will be the voting model. But still, seems to work relatively well in lower k's.
@Dikran, I guess we understood each other well and should stop abusing the comments section, but one last remark: I subjectively choose to accept H1 if I *subjectively choose* to reject [amoeba-reject] H0 based on my *subjectively chosen* $\\alpha$ based on my *subjective* expert assessment of P(H1). Saying that "I am not compelled to accept H1 just because I can amoeba-reject H0" does not make any linguistic sense. But I do agree that "I am not compelled to accept H1 just because I can reject H0 at 5% level". My main point: **being able to reject H0 at 5% level** $\\ne$ **rejecting**.
It is interesting that in some answers, and in many comment threads on this site, people have argued that certain non-parametric distribution tests (such as the Kolmogorov Smirnov) are *too* powerful.  (I don't think anyone has yet argued, though, that such tests are "rarely practical.")  This appears to be just about the opposite of the conclusion you draw.
I planned to estimate lambda for each store/SKU separately.   (1) What did you have in mind regarding mixed models, and what are mixed models :)? I must mention that I'm dealing with very different stores (from 50 sq m to 1500 sq m) with very different turnovers. this is reason why I looked each store separately. (2) I planned to adress this problem with modelling sales per hour. You asked me about my statistical background Well, I have some basic knowledge about mathematical statistics (common distributions, statistical tests, etc..).
I think you switched up pi and mu?
It does not seem reasonable to assume otherwise. We derive the greater set from $S_n$ with no mention of different data sets. By the way, you are right in noting that I have ommited certain parts of the proof. It is only because that very part I present above gives me trouble.
You should probably make your post a community wiki since there isn't a correct answer.
@Aniko thank you to confirm! That's what I was trying to understand in my question comments as well.
Which naiveBayes do you use? This from `e1071`? What is your `housevotes_test`?
Hello again @tester1234. I'm trying to perfomr the Two-Way Anova test for my data. Based on the link you suggested me: which should be my two factors? May I say: which variables should I put inside the 'lm()' function? I have precipitation from 3 stations (pbani, pplc, pra), and I want to know which one is best correlated with a fourth one (psjo). I have monthly data that is grouped in 5 years period intervals (data can be downloaded from this [link](http://www.geografiafisica.org/wp-content/uploads/2014/03/sjo_bani_ra_plc_5yr.csv)). Thanks in advance
Hi.  You're more likely to get a useful answer if you indicate what efforts you've made to solve the problem yourself (eg which definitions have you looked at), what is puzzling you about them (so we can help!), and what the context is (eg regression, time series, multivariate analysis).
Please don't crosspost [questions from other SE sites](http://math.stackexchange.com/questions/635453/probability-distribution-satisfying-constraints). As the [help explains](http://stats.stackexchange.com/help/on-topic): "*cross-posting is not encouraged on SE sites. Choose one best location to post your question. Later, if it proves better suited on another site, it can be migrated.*"
I've added an example. If `x` is omitted, only in-sample results can be computed as explained in the help function. It would be possible to add an in-sample version of Theil's U as shown in the example, but I've never needed it and I can't imagine why anyone else would. But I can add it to the next version if it is useful to someone.
Please don't make assertions about what I think, Tim, because they might not be true. In particular, I haven't made any assumptions about what times are possible. *You* need to clarify what you're asking about, though: if you insist that your time series are actual data sets then, as you say, it's a triviality that $n-1$ differences will reduce them to nothing. But that's not what stationarity is about: it's a *theoretical* construct and applies, *sensu strictu,* only to mathematical models of time series, for which we may assume that $t \\in \\mathbb{N}$ for instance.
Notice that there's a very obvious typo (an extra mathematically ungrammatical '=') in the paper's version of Eq. 12, which increases my posterior probability that the mistake in Eq. 13 is also a typo.
thank you very much and I will take a look "high performance computing in R"
Good answer bru.
Quick and dirty: `=MAX(($B6:$B18=1979)*IF(ISTEXT(E6:E18), 0, E6:E18))`
How do low p-values demonstrate multicollinearity?  The two concepts are completely different.  With enough data, you can have low p-values and almost no collinearity at all.
One thing I had in mind is that PCA generalizes easily to more than two variables. Another is that it treats $X$ and $Z$ symmetrically, whereas your proposal appears arbitrarily to single out one of these variables.  Another thought was that PCA provides a disciplined way to reduce the number of variables (although one must be cautious about that, because a small principal component may be highly correlated with the dependent variable).
To clarify: are you asking if there are statistical tests that address whether or not two samples are from the same distribution?  I.e. that $X$ (larger sample) and $x \\subset X$ (subsample) are similar (or, that $x^C$ and $x$ are similar)?
The R code is open source and you can look at it.
what does about zero mean? Generally when speaking of moments, the tail-behaviour is much more important than values around zero, so it is confusing to see both of the concepts used in the same sentence. Note the both the answers provided are about n-th moments, with no other conditions attached.
Do it the same way you did M, only instead of looking at nearest neighbour transitions, (say, sequences AB), look at pairs that are 2 apart. So, if a subject goes ACB, that counts towards your AB transition count. So does ABB. Create a matrix where item in row i, column j contains the i to j transitions. Then divide by the column totals. You want the columns to sum to 1. Under the Markov property, this matrix should be close to $M^2$
Be aware that asking for code ( / how to do it in R) is off-topic for CV. The other parts of your Q are on-topic, but you may not get an R code tutorial.
(And, IIRC (it's been a few years) the curve we were fitting to the data was the "right" curve, according to theory.  (This was back when I used to test rubbers for the Air Force.))
You may want to post some code and maybe the top of your data. You may also want to do something about your 0% accept rate. That will make people more eager/able to help you.
This question appears to be off-topic because it is about statistics
Ahh, I see..Well there are a couple of things people do. In a lot of the cases they chose what "small" is supposed to mean and filter of at that (0.05 etc.). A method I prefer is to do a sparse-low rank reconstruction and then keep whatever entries remain
For the record, this new definition is $D=\\bar{\\hat\\pi}_1 - \\bar{\\hat\\pi}_0$, which is the mean predicted value for the $1$ responses minus the mean predicted value for the $0$ responses. It can range from $0$ to $1$.  Tjur does not dismiss the Nagelkerke pseudo $R^2$, but suggests it lacks the "intuitive appeal" enjoyed by $D$.
This should be migrated to CrossValidated stats.SE website. I'll wait for it to be migrated, but start reading this in the mean time: http://www.citeulike.org/user/ctacmo/article/8898414
I think it's been asked before, but it's a good enough question.
One quibble. I don't think you can think of them as probabilities when $y$ is continuous.
$\\Phi'$ is of course the derivative of $\\Phi$
Looks like MATLAB to me. "Say what programming language you're using." http://meta.stats.stackexchange.com/questions/1479/how-to-ask-a-good-question-on-crossvalidated
Thank you for your input. I don't really want the number of proportions to affect the p-value. I don't want to "consider" a subject, just because they have more proportions within them, so therefore more opportunity for them to rather be significant. I'd like to nullify this. As of now, I'm just using prop.test in R, and yes I would like to know how to conduct these tests properly. Thank you for your input!
@Behacad Hello, If you look at the equations you see, that for the two class formulation and the general formulation, there are very distinctive comments. Then, at the end of the Accuracy there is a underlining "^^^^^^" where it is specifically mentioned why the equations are not equal. Wish you a nice day.
It seems you know a model for how such a probability distribution would arise. Can you use that to help you?
Actually nothing you say here (or in response to Felix's comments) directly conflicts with what I say. Using change scores doesn't 'adjust for baseline', it controls for any time invariant ommitted variables (or if selection into treatment is highly correlated with baseline). If baseline is non-neglible (i.e. it has a direct causal effect on the outcome or it has an interaction with the treatment) change scores don't solve the problem.
Yes but, according to the code above, you have not applied the centering and scaling to `testSet`. `predict.train` does that but `predict.randomForest` does not.
This looks like a homework question to me. If this is true, could you please add the [`self-study`](http://stats.stackexchange.com/tags/self-study/info) tag? Thank you. Why do you think your result is misleading?
Could you provide some more details about where you're getting stuck? Matlab has functions for several machine learning methods, including the ones you're interested in, and they're pretty straightforward to use. The crossval() function is mostly just for convenience. You could also manage it yourself--it's just a for-loop.
I have no idea why people are voting this answer down.
After a little more reading, I'm wondering if the subtraction of variance in the measurement mean (my question #2) is something akin to Bessel's correction, where the squaring results in an underestimate.  That is, does the measurement error introduce an overestimate bias in assessing the variance due to a particular component?  I'd appreciate it if anybody can shed some light on this.
Yes, mixed models can deal with such designs. But repeated-measures ANOVA does so as well. However, the natural question in such a design is whether or not to center the continuous predictors (probably a good idea). This question is equally present on whether or not you use RM-ANOVA or mixed models.
Thanks Reed. k-means is definitely a way to go and I explored several solutions. However I am trying to go for a solution without subjective decision of the number of clusters.   Hierarchical approach on the other hand produced huge classification and again it was hard for me to specify where to stop.
This will make the final correlations difficult to determine. What you describe is essentially generating from a [multivariate Gaussian copula](http://en.wikipedia.org/wiki/Copula_%28statistics%29#Gaussian_copula).
As for the technical question: You can set the contrasts before calling the model-function by writing something like contrasts(PedonsTx_Reread$Physio.Code) <- contr.sum
Thanks. So let's say that the $K$ functions for processes $x$ and $y$ are given by $K_x(t)$ and $K_y(t)$. How do I test the hypothesis that $x$ is independent of $y$? Under the assumption of stationarity I can simply look at $L_{xy}(t) = \\sqrt{K_{xy}(t)/\\pi}$ and look for departures from $L_{xy}(t)-t=0$. For non-stationary distributions I see that $K_{xy}(t)$ will in general be different from $K_{yx}(t)$, but I don't see what the impact of this on my hypothesis testing should be.
You may read [here](http://stats.stackexchange.com/a/30724/3277) that this formula reduces to the formula of the [cosine](http://stats.stackexchange.com/a/36158/3277) similarity, and _r_ [is the cosine](http://stats.stackexchange.com/a/22520/3277) for centered data.
Ok I think I got it: I just have to keep tallies of how many trials took less then or equal to $k$ times
I'm afraid you're incorrect - a linear model can over-fit the data. You have 30 observations and 18 predictors. That is less than 2 observations per predictor!
Yeah, good points.  But what about common post-treatment trends?  I mean, I guess that I'd have to assume that the early-treated have similar potential outcomes to treatment as later-treated.  But that seems implicit in DinD done the normal way also.
@Andy Could you explain what you mean by "formative measure" and "available information"? // I should point out that *many* measures of soil suitability for agriculture are not even monotonic, much less linear: for instance, a plant might flourish within a range of pH but suffer with pH's beyond this range *in either direction*.  It would be a special circumstance indeed--maybe one involving a narrow range of values--if a simple linear combination of soil characteristics had any objective relationship to agricultural qualities.
@Srikant Vadali $Thank\\:  \\, \\, you$
(1) What's your model to be used for, & what's the goal of selection? - this determines what "best" means. (2) Do the models contain different nos free parameters? - note that the area under the ROC curve isn't penalized for model complexity. (3) How are you measuring cross-validation error? - avoid measures based on classification using arbitrary cut-offs. (4) If you use any response-based criterion, picking the "best" from 16 without shrinkage introduces bias into the predictions & coefficient estimates - you can estimate how much by cross-validating the *whole* selection process.
Well take a look at the red cluster.  It's very well separated if you consider `Petal.Width` against `Petal.Length` but less well separated if you consider `Sepal.Width` against `Sepal.Length`.  You can tell this by looking at the univariate density plots, the red curve overlaps much more the blue and green curves when you consider `Sepal.Width` then when you consider `Petal.Width`.
@onestop I initially suspected that, but then it occurred to me that in the finite case there must exist either an absorbing or a recurrent state; neither of those can be transient.  Thus a finite transient Markov chain does not exist.
what are datasets ?
Thank You! @Stephan Kolassa
I don't follow from very early on. How is generating posterior data standard Bayesian stuff? Bayesian analysis involves priors on parameters or hyper parameters a data set and then a resulting posterior distribution for the parameters.  There is nothing that we would call posterior data. I guess since you have a posterior distribution for the mean and covariance you could draw "samples' for the parameters using the posterior distribution.  But why would you need that?
Not seeing how this answers the question...  which was whether it's incorrect or not to use the first difference instead of the log to compute covariances, not about how to do a conversion.
It might be of interest to note that the logarithm of a variable with this distribution would be a doubly truncated Exponential variable.