tttjjj/elt_bench huggingface dataset:
Silly me.  Just in case someone else is wondering about this, recursively calling the cumsum function for each dimension is the answer:
CDF = cumsum(ePDF,1);
CDF = cumsum(CDF,2);
CDF = cumsum(CDF,3);
I edited your LaTeX to code-format (LaTeX mathematics, among other things, insists on making " <- " into "< -", which is quite a different thing in R). If you want to change it back, you should be able to revert without making the individual changes.
If by hindsight accuracy oyu mean the resubstitution estimate of error rate, it is probably at least partially due to the well-known large small sample optimistic bias of the error rates.  The true classification error rates are probably a lot higher than 20%.
Can you clarify a bit more?
Thank you very much for your kind help.
"The more folds of the resampling procedure we use, the more we can reduce the variance of the estimator": I'd have said: the more we can reduce the variance of the estimator that is due to model instability. The variance that is caused by the finite number of *distinct* cases tested cannot be reduced by this strategy (at least for iterated/repeated k-fold CV).
Or could you at least `dput` the list of `kmeans` objects? Maybe with a reduced number of observations, in order to not let the output become too large.
Thanks, this is exactly what I had trouble with!
Can you post the graphs and/or give more specifics on how you made them (eg did you use the graph builder or were they generated with a different JMP module)?
Part one [here](http://stats.stackexchange.com/q/44840/930).
"right or wrong" requires only a one-word answer, which by the usual policy on answers (that they be more than a sentence) means none of your questions of this form will be answerable in the answer space. Please at least *explain* your reasoning (not just the result of it), and try to phrase your question in such a way that a response can be more than 'right' or 'wrong'
**PART 1** First of all, thank you for your answer. Indeed, frequentist methods correct for multiple testing. Hence, the problem of biased treatment effect estimate cannot come from there. At an interim analysis, the test is based on the current information, using the current sample size, not the overall planned sample size. So the problem does not come from there either.
There are some papers that discuss this possibility (some by Rayner and Best) as well. If you can't find any I should be able to dig some up.
@user24534 A factor is R's terminology for a categorical variable. You have to indicate to R which variables are factors if it doesn't work that out for itself (and it won't for numeric data).
+1 for "The average human has one ovary and one testicle" (& also for the rest of the answer ;-).
I'm not sure that topic modeling is quite what you're looking for, but it seems like it ought to at least be on your radar if you're working on text classification problems.  [David Blei has some good resources](http://www.cs.princeton.edu/~blei/topicmodeling.html).
@Peter See my updated response.
Could you add a brief summary of this paper?
I do not know if there is meaning to hold out of sample data in Croston method. as far as I know, croston and syntetos boylan forecasting had to be evaluated everytime there is a demand. therefore, evaluating the forecast by holding out of sample data seems strange to me.
Thank you for you answer. Your answer is similar to that of Henry. In the same way, I'm wondering if it's not possible to use a k-fold CV for the evaluation, and not for the tuning? Maybe, there is a risk to do that with my proposed procedure?
Therefore, by using the fitted risk (as you've calculated) divided by the time denominator, you will have an estimate of the hazard function. How does it translate to code in r? hazard/t if t equals time?
Your understanding is correct.  This is one reason Normal variables are so popular, both theoretically and in computer models! Nevertheless, the basic idea of looking for a way to transform the variables into sets of independent (or easily related) variables will carry over to a more general setting.
The procedure you're following is unclear because it lacks critical details. I have to deduce that the mean of your multivariate normals is $(0,0,0)$, for otherwise there would be "covariance." But what do you mean that two (*empirical*) distributions "continue to be perpendicular in the x dimension"?  What do you mean by "identical" when you're really talking about two samples? BTW, the theoretical answer to this question is clear: because PCA has an inherent geometric interpretation, changing the coordinates via rotation will not change the result; it will only change how it is expressed.
@Azeem But The degree of freedom is not $k-1$. It is $b-1$ so written on the book.$b$ = number of blocks & $k$ is defined above.
I agree that the background option is a good idea.  But the red / green is not - my office mate is colour blind, and can't tell the difference.  So even though it's good from the "traffic lights" point of view, it can cause problems!
@DTRM - Click the [partition the variance](http://udel.edu/~mcdonald/statanovapart.html) link in the "Model II anova" section of the page I linked you to.
Check the reference and Stata Journal website associated with it.
@ttnphns
1) If I understood correctly I cannot have latent var.with only 1 observed variable. So whenever using CFA I should include only latent variables having more observed variables.

Not completely sure about 2) and 3) can you try to further explain.
(+1) Perhaps worth mentioning that throughout "estimate" means "estimate by maximum-likelihood".
I think we need more information. My post [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) may help.
I am the original author of the `getSummary.mer` function. The reported $p$-values should only be used as a quick check. If I recall, I actually only included the $p$-values to make it work within the framework provided by `memisc`. But this should really be provided with an appropriate warning to the user, and I will contact the package maintainer to see about getting this added. My advice is to follow that provided by Doug Bates: MCMC is the safe bet (assuming others don't have better options).
Other items worth mentioned: **(a)** Recovery of moments via differentiation, **(b)** the fact that all distributions have characteristic functions (as compared to moment-generating functions), **(c)** The (essentially) one-to-one correspondence between distributions and their characteristic functions, and **(d)** the fact that many relatively common distributions have known characteristic functions but no known expression for the density (e.g., Levy stable distributions).
Thanks for the information.I got your point. But I am not really sure how to make user visualize these regime shifts. And yes ofcourse certain hours of the day have unique profiles such that when day start, most of the time incoming documents are more.
I am still not sure how to visualize such scenarios to the user.
Why did you split your data into 3 subsets & fit 3 different models? What are you trying to do? Can you provide any information about your situation, your data & your goals? What do you mean by "the anova code"? Are you referring to the `anova()` function in R?
Yes, it is not a regression plot. It is normality plot. If the dots are on the line it shows that the distribution of the residuals is normal. Distribution of the residuals, not the distribution of actual data.
I like your idea of some form of multilevel model for this problem. I'm not sure if the data can support it though. While many athletes did report multiple injuries, the most any athlete reported was five and this did not occur very often.
The t-test and the z-test are not effect sizes. the same effect size will yield substantially different t and z values for different sample sizes.
Thanks! I think that explanation is good enough ... I will try it out and see how it goes. By the way is there a some code that implements these kind of calculation in python? Thanks
The problem with the last example is that a much better fit is obtained by a function of the form $y = \\frac{\\beta_1}{1 - exp(\\beta_2 + \\beta_3 * x))}$ (notice the negative sign). Is that an acceptable solution?  If not, you will continue to have problems, because you will have to accept solutions on the boundary between the two models (and those are awful).
+1, did not know that you can pass `[[` as a function.
@gung, i think that he's struggling with the fact that side is not ordinal, unlike other variables, but i agree that it could be added to ANOVA as a dummy
At the point where you said "Do you know how can I plot things like this in R?". It is a bit easier to answer questions that are smaller in scope, saying things like "How do I do X?" is difficult to answer besides to point to literature. It is best to show what type of research you have done so far, and if you are getting stuck at a particular point ask about that.
I agree that this is too broad to be answered at the moment. Unfortunately, the CPAN page makes it clear that there will be no support and that users are expected to be familiar with OLS estimation. Take a look at the section related to multiple regression in [The Little Handbook of Statistical Practice](http://www.tufts.edu/~gdallal/LHSP.HTM), and if you are still in trouble with a particular aspect try to narrow down the above questions.
I do not see it as confused. The view I am presenting is common in the methods lit, and reflects desires to treat the number of factors as fixed and independent of subsequent rotation, interpretation, etc. PCA and CFA are not equivalent, and different perspectives hold sway as to the appropriateness of either in selecting #factors actually used. Last two questions: empirical *retention decisions* are not full on, and the use of MLFA either requires the retention decision a priori, or entails different motivating assumptions about retention.
Welcome to CrossValidated! I suggested an edit to your post that changes the formatting a little; feel free to roll back the changes if you don't like them.
I am interested in the result of your experiment!
@drknexus In typical team studies, each team member is asked about for example their level of agreement with statements like "my team is performing well", "team members get along well with each other", "my team handles conflict constructively"... The question is thus the degree to which such beliefs are shared by team members.
Good edit. But that now changes your question - where does your problem currently lie? Or is it solved now?
In that case, the Wilcoxon is not the ideal test to use. For each participant you have 36 or 72 data, ie >2. When you reduce your data to 2 points by averaging, you are throwing data away. You would have better power by using a GLMM or GEE logistic regression (my answer here: [Difference between generalized linear models & generalized linear mixed models in SPSS](http://stats.stackexchange.com/a/32421/7290) discusses the difference).
It will not solve your problem but http://stats.stackexchange.com/questions/2492/is-normality-testing-essentially-useless might be relevant.
On the other hand, my answer is derived from reading your paper/presentation in the past.  So I guess I'm copying you as well.
You have been given enough information there to answer the questions. See of the discussion of sequential sums of squares [here](https://onlinecourses.science.psu.edu/stat501/node/69), and note how proportion of variation explained relates to regression sums of squares [here](http://en.wikipedia.org/wiki/Explained_sum_of_squares)
Any application in mind?
Adding predictors that explain some variance can improve model fit even if those predictors don't have clear, independent effects. Adding more predictors that aren't completely unrelated to the outcome can also decrease the amount of outcome variance that covaries uniquely with previous predictors. Are you hoping to understand relationships between your predictors and outcome, or is predicting the outcome the only major priority?
As it stands this is more suitable as a comment than an answer. Could you give an outline of the algorithm here, please?
*Why* are you building this model? Do you want to be able to predict something in the future, or are you trying to understand why something happens, or do you simply want to describe the strength of the relationship b/t some variables, or something else? What you should do depends critically on what your goals are.
However, even when standardization is applied (and therefore mean centering) on the predictors in the algorithm, they take the unstandardized data (therefore $\\bar{x} \\ne 0 $, in general) to fit the different intercepts displayed by `glmnet`. They do use the same $\\beta_0=constant=\\bar{y}$ for updating the coefficients but fit an intercept with the raw data, and they do it a posteriori.
Wow... that software seems extremely powerful. I guess that answers the question that it is indeed $O(1/n)$ too. Thank you very much.
The odds ratio sound like a good idea. If we let $G$ be logistic function, $G(x) = \\frac{1}{1+e^{-x}},$  $Y_{st} = G(\\alpha_{0} +\\alpha_{1}R_{st} + \\alpha_{2}I_{st} + \\delta_{t}+ u_{st})$ be a transformation of above model, then if we estimate $G^{-1}(Y_{st}) = \\alpha_{0} +\\alpha_{1}R_{st} + \\alpha_{2}I_{st} + \\delta_{t}+ u_{st}$ (this gets rid of ceiling effects since $G^{-1}:[0,1] \\to \\mathbb{R}$) using OLS, are the estimates here odds ratios? Because I would like to transform the model to eliminate the ceiling and floor, and still be able to interpret the estimates in terms of odds ratios.
Both, in their own context. BMI is well understood and is an easier "sell" in discussing your analysis. But if the groups differ in a way substantially different than the BMI, the manova will pick up on that (provided it is linear). Another idea is to take logs. Note that log BMI = log wt - 2*log ht, so if you do the multivariate analysis on the logged responses, and see that the coefficients are more-or-less proportional to 1 and -2, that'd indicate that BMI is the big player and you could simplify. The log scale may make the diagnostics look better too.
You quoted the abstract of a paper and provided something that resembles a URL (I took the liberty of fixing the link). That's not really the type of answers that we are looking for here; I encourage you to edit your answer and to try to add some comments of your own about why this link helps answer the question. The answer would be much improved if you explained how this asymmetry index is related to the mean central tendency and MAD.
Are these lists of *words,* of *phrases,* *sentences,* *paragraphs,* *entire documents,* or something else?  I think the answer is crucial.  Consider, for example, an extreme case where corpus 1 consists of ten documents and corpus 2 of 50 documents.  Perhaps one of the ten in corpus 1 mentions "forest," occurring 20 times therein, and five of the 50 in corpus 2 mention "forest," occurring from 20 to 150 times per document for a total of 300 occurrences.  Now it appears there is *no* significant difference in usage of "forest" between the corpora.
PCA software worthy of use will standardise for you on request. Doing it yourself is pointless.
Your question is directly answered in the part you quoted. Start with the fact that a parabola [could be written in a vertex-centered form](http://en.wikipedia.org/wiki/Parabola#Equations), and then read to the end of the sentence from "**equating coefficients**, we find ...".
Charles is right: I'd give it a try with `nls`. The crucial aspect of using `nls` is to provide sensible starting values. An example code could look something like `nls.mod <- nls(Wa~Wma*(1+a*(Na + alpha*N))^(-b),  data = dataset, start = list(a = 1, b = 1, alpha = 1))`.
How? If I use a logit link function I cannot have low probabilities for low values (]-10 ... -5] ) and high values (]5...10]) but high probabilities for medium values (]-5...5])?!
spectacular.  Thanks
Is this more of an R / programming question? I can't tell for sure. If it is, it would be better asked, & probably answered, on [Stack Overflow](http://stackoverflow.com/) rather than here. If this is a question primarily about statistics, please edit to clarify, OTOH, if you just want to know how to get R to do this, flag your Q & we can migrate it for you (*please don't cross-post, though*).
Normally in statistical estimation problems a sharp distinction is made between variables with a hat and those without. Subscripts matter, too. Are you making such distinctions between $\\theta$, $\\widehat{\\theta}$, $\\theta_p$, $\\theta_a$, and $\\widehat{\\theta}_p$ and, if so, what are they?
Is that the "Riemann manifold Langevin" paper? Do integrate Fisher information at some point?
In fact, +1 for the clarity and usefulness of that additional context; sorry if this makes for conflicting feedback from me.
I would love to see an example, even a contrived one, of estimating both $n$ and $p$ (in a frequentist setting).  Think about it: you observe a *single* count, *k*, say $k=5$. We expect $k$ approximately to equal $n p$.  So do we estimate $n=10$, $p=0.5$?  Or maybe $n=5000$, $p=0.001$?  Or almost anything else? :-)  Or are you suggesting you might have a series of independent observations $k_1, k_2, \\ldots, k_m$ all from a common Binomial$(n,p)$ distribution with both $p$ and $n$ unknown?
Check our rapidminer so that you do not need to do manual coding. If you can structures the data, it is quite "easy" to apply a learning algorithm to it. Naive Bayes is one option but you might also consider SVM, k-NN, Decision trees and so forth.
@StasK: I think that the suggested calculs of DF does actually allow for same-coefficient-different-covariance. On the other hand, I am pretty sure a naive run in Stata will indeed not allow for a different covariance along the periods.
As a linear combination of the old one. For example, if you run PCA on color pixels (RGB) of natural images, one component that you get is usually aroun $3^{-1/2} (1,1,1)$. When you think about it, this new components represents the dark-light feature, because it does not favor any color and the brighter the pixel is the larger the projection on the component will be.
@B_Miner: To expand briefly on Henry's nice answer, the formulas you provided in the question are completely general in the sense that they hold for *all* distributions of nonnegative random variables.
OK, I see. Although it won't help you directly with your problem, I don't think you should use this technique as the equal variance and mean and variance are independent assumptions are likely not met. An alternative might be to use a (quasi-)binomial model if you know the successes and trials or a Beta model if you have only proportions.
many thanks hadinbe, that clarifies things for me!
If you could work out the joint probabilities then you could work out the conditional probabilities.  So for `P(D=true|A=true,S=true)`, it would help to know  `P(D=true,A=true,S=true)` and `P(D=false,A=true,S=true)`
@whuber @Gaetan's question assumes that one study is closer to the truth. I try to take a step back and situate variations in results between studies within a meta-analytic framework, acknowledging the possibility that the studies may be of equal quality, but that random sampling or substantive differences may be the explanation.
To say the Deducer GUI to build plots (via ggplot2) is interactive in this context is misleading. It allows you to choose elements to add to the chart (and update the chart if need be), but it doesn't allow you to interact with elements on the chart (e.g. brushing/linking).
Note that $2^m=n+2-k$, then the convergence of the series is not ensured because the summation is over $n$. Otherwise, you would be correct, [see](http://www.austinmohr.com/Work_files/prob2/hw1.pdf).
Not sure how you build a confidence interval from this matrix of n.bs prediction intervals.
I found this CV thread, http://stats.stackexchange.com/questions/63566/unexpected-residuals-plot-of-mixed-linear-model-using-lmer-lme4-package-in-r, to be very helpful. The post explains how to create a binned residuals plot in R.
I should clarify- binomial models only work for the case where $P=0$ or $P=1$. If there are intermediate values, you should stick with least squares.
(ctd)... Your data will almost certainly be neither exponential nor gamma (George Box's epigram is relevant); the relevant question is not whether you can *detect* a difference (as sample size grows, this becomes a certainty) - it's *how much that matters* to your inference. That's not a question answered by hypothesis tests; it's better addressed by diagnostic tools like Q-Q plots and simulations.
Couple of books that you might be interested in "What if there were no significance tests?", edited by Harlow, Mulaik and Steiger, and "The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives" by Ziliak and McCloskey.
I'm using the same alternative hypotheses, and I've checked so that simulations will actually converge towards the specified means and standard deviations.
I think Bayesian statistics as my Digital SLP camera while frequentist as iPhone camera. I bought both of them sometime but I use DSLR less 5% of photos while phone rest 95%. Because it is easy, handy and in pocket and many time provides as per quality (based my DSLR skills). Just like incorporating the priors and running chains in hayes I need to find optimum balance of aperture opening duration, length and other parameters. Iphone end of popular.
Jennifer, sorry. Your question is primarily about programming of code request, its statistical tunes are muted.
+1 for stripping down the desired result to the essentials. I will add that for the more general case of joint normality with unequal variances, a rotation of axes by $$\\theta =  \\frac{1}{2}\\arctan\\left ( \\frac{2\\rho\\cdot\\sigma_1\\sigma_2}{\\sigma_1^2 - \\sigma_2^2}\\right )$$ instead of the $\\pm \\frac{\\pi}{4}$ implicit in $(X_1,X_2) \\to (X_1+X_2. X_1-X_2)$ produces independent normal random variables.
For what it's worth, I'm not sure greedy algorithms are necessarily sensitive to initial conditions or always miss optimal solutions. My impression is that some min-spanning-tree algorithms and Huffman coding are both greedy and optimal. Still an interesting question though!
[The Wikipedia entry](http://en.wikipedia.org/wiki/Cauchy%E2%80%93Schwarz_inequality#Probability_theory) for C-S in probability theory seems to have what is used in the proof above: $(\\mathbb{E}XY)^2\\leq\\mathbb{E}X^2\\mathbb{E}Y^2$ or $\\mathbb{E}XY\\leq\\sqrt{\\mathbb{E}X^2}\\sqrt{\\mathbb{E}Y^2}$
@gisol: Welcome to stackexchange!  It's preferred to flag a question for migration rather than reposting. Just click the flag link and write the moderator a note.  Also, for borderline questions like this, the consensus usually is to leave it where it is and see if you get a response; if not, then ask to migrate.  But the community gets a say too; if enough people agree with David Robinson, it will get migrated without you having to do anything.  One more note; most of the same people who answer questions like this hang out at both places, so migrating usually doesn't help get more answers.
Thank you for providing such a clear explanation.
Very nice call-out of the underlying assumptions (+1).  I agree with most.  Do you have any references to papers or methods that deal well on real-world data and address any of the six listed issues?
This is excellent. Thanks for all your work! I assume instead of using a pre-programmed parametric log-likelihood function for the noise and estimating its parameters I could also try to estimate something more general, e.g. fit a KDE?
Not clear how far this is a general question and how far a highly specific question on how to use scikits for the purpose.
Because the Beta distribution is undefined when either parameter is zero, it is not possible for its parameters exactly to have either of these distributions (which place nonzero probability on zero).