tttjjj/elt_bench huggingface dataset:
Dear Gung, Tks for your note.

Sorry, letter (b) is another question. (If "Y" .....) are information from the original question and not from de letter (b) that will be answer after (a).

Bit error probability .... is the beginning of my reasoning  
"e" = "and" 
about tag that´s ok.

tks
@Irish I don't follow that.  T1 doesn't seem to be a lag: it stipulates a changepoint.  Isn't it easy to find "significant" changes if you're allowed first to inspect the data to choose where the changepoint is?
Thanks. That's a nice article. It's a start at what I have in mind, but as you say it's very brief.
Thank you very much. Your answer helped mr but it is only a part of the answer I am searching for.
I improved it by choosing the mean of the 5 "closest", but it is still not near the classification tree. (closest: 0.68 error, best of 5: 0.72 error, 0.77 is classification tree).
@amoeba: I think you may have misunderstood my answer (and possibly the others you downvoted as well). I did not talk about modeling spirometry results as a DV, but as an IV - with the challenge that the measurements are correlated for each participant. My suggestion was that the OP should not average the three measurements and use the average as an IV, but to use all three measurements as an IV and account for the dependence between each participant's multiple measurements using mixed models. Please reconsider your downvote on my and the other answers.
Would more usually be described as a distribution _conditional_ on $u–v \\ge 0$.  'Prior knowledge' would be of a parameter.
How, specifically, do you propose to test *distributions* using "t or F statistics"?
Thanks! Isn't ordinary least square assuming the covaraince matrix is diagonal and the diagonal entries are the same?
**First guess**: You're a C/C++/Java programmer used to zero-indexing. Matrices in $R$ aren't zero-indexed, but rather one-indexed. Also, you use the call `V(g)[i-1]` which the first time through the loop is `V(g)[-1]` and the second time is `V(g)[0]`, neither of which, I'm guessing, are what you actually intend to do.
@Zach - After the command "DT <- DT[,list(n=sum(n)),by='hour, minute']"
 this R beginner got an error message:  "Error in eval(expr, envir, enclos) : object ' minute' not found".  Any idea why?
Give me a moment; I was just reading the original papers to check I hadn't missed anything. .... and now finished
Have you obtained explicit formulas for the marginal distributions?  If so, they will tell you what the joint distribution of $(X,Y)$ should look like *assuming* independence.  How does that compare to the distribution you have?  Alternatively, can you write $P(x,y)$ ($n$ is not an argument because it depends on $x$ and $y$) as a product of a factor depending only on $x$ and another factor depending only on $y$?
(ctd)... Note also that if you make your histogram bins of width $h$ that will scale the height by $1/h$, so you also have to adjust for that effect (but seriously, don't use histograms for discrete data).
Depending on your model, there is a logic to this in so far as the different subseries could be considered as realisations of the same generating process. Depending on your data, I would guess wildly that this won't work well because of side-effects of the subseries not joining smoothly. Aren't bull and bear both nonstationary any way (not my field at all; correct my ignorance)?
"enough" means that someone will answer in a way that satisfies you; that's doubly difficult to predict. As @whuber advises, do look at what's already on-site.
Thanks @user155437 for your help. But, I'd like to ask you:

1. How did you derive $\\hat{\\psi}^2(Y_j)-\\psi^2(Y_j) = O_p(1/\\sqrt{m})$? Since using Delta-Method, I was only able so show $\\hat{\\psi}^2(X_j)-\\psi^2(X_j) = O_p(1/\\sqrt{m})$.

2. Once, (1) has been proved, how will we make sure that the sum is also $O_p(1/\\sqrt{m})$?
Welcome to the site, @raid amin. I am not sure there is enough information here for the OP to be able to conduct these analyses, If they do not use SAS. There are some other alternatives, including permutation tests & ordinal logistic regression (as discussed in the thread linked above). On a different topic, please do not sign your posts, CV automatically attaches your username, flair & a link to your user page to any answer you post. If you like, you can display your email address there.
(+1) A couple other simple examples to consider to see the wide disparity of information available: (**1**) $X \\sim \\mathcal U[0,1]$ and $Y \\sim \\mathcal U(1,2]$ with $f(x,y) = x$ and (**2**) $X$ and $Y$ with arbitrary distributions, but $f(x,y) = y$.
How to perturb the parameters? Assume you have your parameter vector p, then you calculate the perturbed vector p_prime = p + n with n being a vector where each component is being independently sampled from a normal distribution, e.g. N(0, sigma).
Incorrect. Inference on the Pearson correlation does not require any of these assumptions to hold. One could have a curvilinear relationship with heteroscedastic, non-normal data and the test of Pearson correlation (which is equivalent to inference on a linear regression model) is powered to detect the first order trend. The interpretation of Pearson correlation as the strength of first-order trend still holds. There are few circumstances where the first order trend measured by the Pearson correlation is not adequate for an analysis.
I think you actually want an interval scale. The fact that you do not want the difference between 72 and 60 to count as '6 times' the difference between 60 and 58 simply suggests that your definition of 'quality' is not a linear function of the attained score.
I have seen this when the linear model was fitted without intercept term with the response that was not zero-centered. In such case the obtained model is compared not with "average-response" null-model, but with "zero-response" null-model.
quite good way to explain the nuances.
Dear anonymous user, please don't use edit for comments (it is not for you, Neil).
@JohnK Quite so.
I found my mistake. Instead of reflecting back AFTER taking the power of my logged variable, i made a subtraction IN the power term. Sorry for wasting your time!
If you expect the error variances to be similar, one regression (with interaction, perhaps) makes more sense, since you can do things like fit parallel lines and more easily test for differences.
Looks like it could be very nice, but I just tried installing and running it in Stata 11.1 (i.e. the latest version) and it keeps giving me an r(3000) error upon clicking "Done" in the dialog, even if i type -version 6: clt-.
Can you explain your situation a bit more?  How large is the parent population?  Also, what do you mean when you say you will use the 5 samples of N=3,000 for "training"?  What or who is being trained?
Apparently it's SPSS.
@guy Those methods are not fully Bayes but yes they do exist: "quasi-Bayes".
**[Renyi entropy](http://www.google.com/search?q=Renyi+entropy)**
That's interesting @Dimitriy, but I feel obliged to point out it's the *reverse* situation: the regression itself is not significant even though individual coefficients are "significant."
Did you try to read http://stats.stackexchange.com/questions/tagged/mcnemar-test topics here on Mcnemar test?
As a non-expert, I think that the key to the entire debate is that people actually reason like Bayesians. You have to be trained to think like a frequentist, and even then it's easy to slip up and either reason or present your reasoning as if it were Bayesian. "There's a 95% chance that the value is within this confidence interval." Enough said.
@Procrastinator The link is a perfect example of what I'm talking about: "Then, if our model fits the data well, the ratio of the Deviance to DF, Value/DF, should be about one. Large ratio values may indicate model misspecification or an over-dispersed response variable; ratios less than one may also indicate model misspecification or an under-dispersed response variable." What's the range of "about 1"? 0.99 to 1.01? 0.75 to 2?
For medium to large datasets your recipe will be a reasonable approximation to the answer, but an exact answer is more difficult to come by. "Normality" typically refers to the distribution of the *theoretical* residuals and not the *observed* residuals (which, among other things, are *negatively correlated* with one another). Moreover, your $z_i$ are not the expected values of either form of standardized residuals (theoretical or observed).
Hi and welcome to the site.  Your question is fine etiquette wise, but as a general comment more specific questions are likely to get better answers.  For example, you ask for downsides in general - do you have any specific concerns about the method?
If this is still an active question for you, perhaps we can work out how to apply what I did for the Binomial example for yours?
ANd I should say that yes I have low number of replicates. only 3 replicates!
Most programmers know "median".  (sort(array))[length/2] is a big enough hint for those who forgot.  Also at its most basic for each new point you only need to do a bisection/insert on one half of the array...
@DavidMarx Given the proximity of the two questions, I suspect that the original intent was probably to simulate to find the result.
related [question](http://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what)
You might be interested in [this discussion](http://stats.stackexchange.com/a/35524/919) in a closely related thread.
It might be not related. Read about Simplicity does not imply accuracy, in A Few Useful Things to Know about Machine Learning by Domingos.
Data aggregated as per your example (i.e. summarised for each age group at each hospital) is all you need for this analysis. If you're doing a summary-statistics analysis, then you can do it directly off that original data (e.g. proportion given treatment in Age Group A in Hospital 1 is 3/5 = 0.6, for same Age Group in Hospital 2 is 7/10 = 0.7.)
Why do you want the data to be normal? What are you going to do with the data?
I am hoping I can help you with this question but two things are very unclear to me.  What do you mean by incomplete?  Are you using the EM algorithm because you need to handle a missing data problem?  By starting values are you referring to the initial guess at the parameter values?  If so it is possible that the likelihood function has more than one local maximum and the starting values in A are far enough apart from those in B that they take you to different solutions.  Also if you have missing data a missingness assumption has implications on how to solve the problem.
@NRH - just to add to this answer, the mean and variance formulas can be derived by differentiating the equation $\\int f(y,\\theta,\\psi)dy=1$ on both sides with respect to $\\theta$ (or equivalently $\\mu$).  First derivative gives you the mean, second gives you the variance.
This is a very ad hoc method but one I've used before: If you have two collinear variables, regress one on the other and take the residuals from that model and using them as a predictor in the original model. This would be loosely interpreted as "the part of the second predictor not subsumed by the first predictor".
Did you add a `class` statement with your categorical variable? If not, SAS is interpreting it as continuous.
One could always construct an initial fitted object with all response values being 0, say, and replace the coefficients in the fit with desired parameter values. Then simulated responses can be obtained using 'simulate'.
Check the site.  I think the @Gung and a few others who are more expert than me on ordinal data have commented on using averages of ordinal data as though it were interval. Maybe Gung or Peter Flom will see this and comment.
Generalized Pareto Fit To The Society Of
Actuaries’ Large Claims Database North American Actuarial Journal Pg 18
Note that $p(x_1,x_2,y_1,y_2)=f(x_1)g(x_2)h(y_1,y_2)$. What does this tell us about the independence of the random variables?
Are the seasons the same length? i.e. same number of days. Anyway, I would create an expectation based on length of the seasons then use a chi-squared test.
@TaylerJones Levene's test is somewhat more robust. Browne-Forsythe is more robust (but loses a little power near the normal). Fligner-Killeen more so again. In several decades, I've used Levene or Browne-Forsythe no more than twice each. (If it came up again, likely something like Browne-Forsythe would suit me fine but I don't generally have situations where it makes any sense to test several groups variances for equality.)
plot(cumsum(pve), xlab="Principal Component ", ylab=" Cumulative Proportion of Variance Explained ", ylim=c(0,1))
where pve = proportion of variance explained.
Comments for downvote are always welcome!
I don't see where sensitivity and specificity add anything over examining predicted probabilities from the combined model.  Plus they are in backwards information order.
@Tumbledown Thanks for your nice words. It is nice to know that you are appreciated.If you need help with daily modelling you can contact me at dave@autobox.com and I will try and pass on pointers to you as to how to proceed and some of the pitfalls you might encounter on the way.
Here's a blog article I wrote about interpreting categorical variables in R: http://anythingbutrbitrary.blogspot.com/2012/04/lm-function-with-categorical-predictors.html. You have choices about how you want to code your categorical variables into dummy vars.
This is a very practical guide, thank you very much for gathering all that information.
Thanks a lot, Aksakal!
@ aplassard.  The first article is very interesting, but totally empirical.  The second, is wonderful.  I think that pretty much anytime I read something by Efron.  For the record, section 8 discusses cv, and in section 9 there are comments explaining the relationship between cv and bootstrap that are really interesting.
If you ask all SME of a single country, then you have taken a census of the population of the country, not a sample from the population of SMEs in the country.
The answers are yes (relative similarities depend on the measure), no (they do not preserve ordering), and yes (clustering does make sense), respectively, because an appropriate choice of similarity measure depends on *what the data mean* and *what the analytical objective is.* For this question to have anything other than this trivial general answer, then, please explain what specific problem you need to solve.
the equation would be $\\hat\\beta=(\\tilde X'\\tilde X)^{-1}\\tilde X'y$, where $\\tilde X=[X,Z]$.
+1 These are very good. Thanks for sharing the references too. Will definitely look them up. Very useful!
linkrot has made the linked data unavailable. It sounds like it was small enough to have been put in the post. Unfortunate. Were the values between 0 and 1 compositional data (a continuous proportion, such as the proportion of some substance in some other substance), or count-fractions?
Hi Maarten, thank you very much for your answer. 
Unfortunatly my explanatory variables are all categorical (some ordered) as well as the latent variables (which form the levels). So as far as i know centering them is appropriate, is it?
I don't really understand it. A multinomial distribution accepts a vector of scalars (let's say [3, 1, 5], where each scalar is the number of successes of a category) and returns the probability that this combination of successes happen. So if I sample from such a distribution I should get a vector [k1,k2,k3].  I am true?
@Aniko I beg to differ: please reread the original question.  That's a three parameter model, with one for the constant, one for $\\log(A/B)$, and the third for $\\log(A/C)$, as listed under "Coefficients" in the output.  In this model, $\\gamma_1+\\gamma_2+\\gamma_3=1$ is the constraint, not $\\gamma_2=0$.
Tikhonov regularized regression (a.k.a. ridge regression) does not map onto a maximum likelihood problem directly - it's a penalized least squares problem. In Bayesian statistics, ridge regression is equivalent to maximizing the ordinary Gaussian likelihood with a Guassian prior on the $\\beta$'s.
I know that it is related to the method of comparison. But in my example which method is more reasonable, paired or unpaired? I cannot somehow decide. IMHO, if I was interested in the effect of the data change on single time series points, I would choose a paired test. Otherwise, if I am only interested in the difference between the means of the 2 time series, i.e. are they equal or not, than I would choose an unpaired test (?)
Since this is homework, it is very important that you show us what you've tried to do to solve it.
Please explain what you mean by "descriptive statistics."  The ambiguity here is that a 95% confidence interval ("CI95%") for any parameter is *not* descriptive, so by including it in your list you suggest that you might be asking about *inferential procedures* rather than descriptive statistics.  The answer to the question is no for one and yes for the other.
It looks like this question should have the tag `self-study` added.
@David this is not just a visualization problem, you may be interested in the discussion of split-apply-combine strategies facilitated by Hadley's plyr package http://had.co.nz/plyr/plyr-intro-090510.pdf
Range does not equal variance Penguin Knight so you might want to update your wording here.
@whuber In your explanation of Normal approximation, the calculations of mean and standard deviation is different with the description in Wikipedia. In Wiki, the mean is np, and the standard deviation is np(1-p). So, in this problem, for the Normal approximation of varying probability of success in binomial distribution, the mean is p1+p2+p3+p4+p5+...+pi, and the variance is p1(1-p1)+p2(1-p2)+...+pi(1-pi). Am I right?
How!? I don't see how it would be any easy to interpret in this case as opposed to when the Principal Components are not sparse.
@whuber Thanks for the reply. Indeed I should reject $H_0$ at the **1%** level confidence. I am confused though - at what level is rejecting better 99% or 1%?
@Xi'an  You are correct, I did not process them as parameters of the exponential family.  Doing so would have made the family no longer a natural family, and including them would have just muddled the algebra for comuting the moment equations (which was muddled enough to begin with).
@Patrick, I don't have Y (response) variable in the original data. But I have like two groups of X variables (like 10 and 40) they contribute to the construction of the first two components, so how to do PLS in this case? All 4 clusters were from 50 genes or the two components.
@whuber It's one of the default functions in SPSS graphical editor, called "Confidence Interval for individuals." My guess is that it's the 2.5th and 97.5th percentile of the residual.
@MånsT: Done, thank you for reminding me.
The 36 number is applicable to just one random variable with poisson distribution (i.e. one chromosome), To consider all 5 it is a bit more complicated (as the answer by @AssadEbrahim demonstrates).
http://stats.stackexchange.com/questions/15168/graphing-effect-size-for-coefficient-of-determination
Dear @Peter Ellis, I am actually disaggregating annual to quarterly. Many thanks for your time!
I agree, there is obviously a better MSE out there, which the optimization process failed to find, probably stuck in a local optimum. But I tried tweaking the hyperparameters, nothing doing, still these sub-optimal predictions...
I think this is due to 'gmm()' function. The general formula of the s.d. of a GMM estimator depends on the weighting matrix and involves some matrix inverse. There might be slight difference due to computational error.
Yes. The problem here is that you don't know what distribution to fit - the L-moment diagram could help you with this, and provide a first estimate of the parameters. I can't see all of your dataset (can you dropbox it)?
@Luna, why is that wrong? It appears you used `x3` to generate the `y`s, so it should be included in the model and the $p$-value agrees with that conclusion.
Eventually was able to estimate this by running a poisson regression to get starting parameter values, then feeding these into a MLE on the log likelihood function. Will post the solution to this soon.
In this case, I don't think that using `&&` makes sense, see `?"&"` and [R: Logical operators: Beware of the difference between “&&” and “&”](http://promberger.info/linux/2008/09/12/r-logical-operators-beware-of-the-difference-between-and/) .
I will just add one more, Yahoo!LDA - https://github.com/shravanmn/Yahoo_LDA
@Glen: That is why I tried to quickly remove my comment about the (un)usefulness of a t test in duration analysis. But you were even faster. Another concern is the grouped structure of the data. But your answer exactly fits to the question.
@carbuke But the mean of the distribution is again dependent on y i.e wDy/(w+ϕ). So how do I get the mean of the distribution of y
The selected answer at that link is somewhat misleading. While you do get many diagnostic tools with RF, you can only get a plot like the one shown if your RF is fitted using only stumps. This is because the plot assumes the model is additive in the predictors, which will not be true if the base trees are more complex.
Actually, Dail, your question does *not* use "mean" in the sense of an average a group of data.  If it did, it would be sublimely Zen-like, because it would be asking "when is 8 (a number equal to the mean of (2,4,12,14)) not equal to 8 (a number equal to the mean of (-20,-10,10,52))?" The correct and incontrovertible answer would be "never: 8 always equals 8."
@not_my_birthday: You're welcome! Welcome to the site! :) Your question is an interesting one, but also a well-known one. Please don't be discouraged by the suggestion to look for duplicates. I hope you'll look around and continue to participate here. Cheers. :)
I'm not sure I understand correctly the OP's question. But if $Y_1$ and $Y_2$ are jointly Gaussian, you should be able to push this a tad further: $cov(Y_1,Y_2)=(\\mbox{Var}(Y_1+Y_2)-\\mbox{Var}(Y_1-Y_2))/4$
@Farrel - here's a very short script, which assumes [0,1]-uniformly distributed covariates, an OR of 2 between the first and third quartile of the covariate and standard normal noise, leading to power .34 for n=100. I'd play around with this to see how sensitive everything is to my assumptions: runs <- 1000; nn <- 100; set.seed(2010);
detections <- replicate(n=runs,expr={covariate <- runif(nn);
outcome <- runif(nn)<1/(1+exp(-2*log(2)*covariate+rnorm(nn)));
summary(glm(outcome~covariate,family="binomial"))$coefficients["covariate","Pr(>|z|)"] < .05})
cat("Power:",sum(detections)/runs,"\\n")
As it stands, your question sounds like "why do we use our data in order to analyze our data?"  Do you see why that is confusing?
I like to see vector graphics (no jpegs), graphical design following the principles of Tufte & Cleveland, readable fonts, uncluttered legends, no shaded backgrounds, sensible axis limits and tick intervals, labelled axes, no overlap of text and plotting characters or lines, etc. Most authors use the default settings of their software, so good software has good defaults. This is where Excel fails miserably and R does pretty well. But it is possible to produce lousy graphs in R and good graphs in Excel. It's just easier to produce high quality graphics in R.
And another suggestion: In general, if you're wondering whether your model/code can perform like you hope it helps to generate some data from your prior and fit it. So here you might take $\\sigma^2$ and $\\sigma^2_\\theta$ orders of magnitude different, generate a bunch of data and see what happens.
After reading the **R** tag info, I agree.  Can the question be moved over or should I add the question there manually?