tttjjj/elt_bench huggingface dataset:
Me personally? Rarely - just whenever I had already computed such a CI and wanted to do a hypothesis test instead/as well. If I knew I wanted to do a hypothesis test (which is not at all common for me), I'd do that. If I knew I wanted a CI, I'd do that. It would be fairly rare for me to want both on the same problem, but when it happened, I could just use the CI for both. More typically I'd want to quote the p-value in a hypothesis test, and a CI for a pre-specified coverage doesn't tell me that.
@hoof That's far from the best you can do, because positing a triangular distribution is just inventing information where none exists. Any subsequent results of your analysis would therefore have this (potentially huge) element of arbitrariness to them, which is rarely a good thing.
I don't know how to reconcile "we don't want to treat `'unknown'` as its own gender value" and "we need to include the `'unknown'` levels as baseline" comments. If the `'unknown'`s are included, how can they be included but as their own category?
@Glen_b it's for my PhD. I've oversimplified the actual problem that I have
A similar question: http://math.stackexchange.com/questions/240164/cdf-of-sum-of-dependent-random-variables . By treating my random vector as a vector of random variables whose joint distribution is known, the answer there may apply. But I'm still hoping to find a general statement of how that sort of solution was arrived at.
Your outcome is the *total* number for a set of duplicates. The exposure variable is the number of duplicates.
It *seems* that you ask a simple question, but it is couched in highly technical language. That language would suggest your question ought to be read with great care and caution to make sure every aspect is correctly interpreted, in order to appreciate any unexpected subtleties. In particular, exactly what is "$f_{nj}$," how does it differ from "$f$," and why are you using pairs of subscripts on it, including one ($n$) which seems to be fixed? Why are you subscripting $p$ sometimes with "$j$" and sometimes with "$\\xi$"? Why is the domain of $f$ merely $[0,1]$ and not $\\mathbb{R}$?
To rewrite sm::hsj I'd also have to rewrite sm::sj, sm::hnorm, and sm::wvar.  This would probably take me the better part of a day to get it debugged and working properly and I don't want to do this if I can avoid it.
Beyer et al. seem to be addressing a little bit different aspect of the NN problem. But, for (binary) classification purposes, under mild conditions, it is a classical result that 1-NN classification has, *in the worst case*, twice the probability of error of the Bayes (i.e., optimal) classifier asymptotically. In other words, the first nearest neighbor contains "at least half the information" about the label of the target as the best classifier does. In this sense, the 1-NN seems quite relevant. (See Cover & Hart (1967) for more. I'm surprised Beyer et al. doesn't cite it.)
This is correct.  Or just use the linear predictor in the call to `rcorr.cens` if you do not have any strata.  Take $D_{xy}$ from the output in that case, negate it, and use the relationship $D_{xy} = 2\\times (C - \\frac{1}{2})$.
Thanks, fixed. I have access to acomputing cluster, so computing power isn't a huge problem, but yeah, even getting the one model I'm using set up properly is a pain, and it's already written. Which is why CMIP exists, so people don't have to go through all that trouble each time. Would be interested ti know what field you're in, if you're doing similar stuff.
I appreciate the answer (and the others), but I'm having a hard time accepting it. If I include a 1:0, and the continuous time variable, I still have to assign the non-breast-feeders a value for time (or else they drop for a missing co-variate). Even conditional on the 1:0 variable, I don't see how including the non-breast-feeders as time=0 doesn't affect the regression coefficient. Perhaps also adding the product interaction term between the two would make more sense?
@Michael Chernick, Since, Logistic seems better classifier for `YES`, whereas ADTree seems better for `NO`. I have tried `Boosting` logistic, it did not worked either.
I couldn't agree more with the general advice to visualize and for this problem the specific advice to consider density estimation. But saying that you shouldn't do cluster analysis in 1D or 2D is more than a trifle dogmatic. People often need or want clusters for scientific or practical reasons, even with 1D or 2D data. For example, if the science says there are clusters, then wanting to test that is a natural thing to do. Considering that there might be a continuum instead is also a natural thing to do. Comparing "automatic" with manual clusters is also a natural thing to do.
I tried solving this myself, gave up, and wrote a follow up question here:  http://stats.stackexchange.com/questions/13721/when-is-the-maximum-achieved-for-the-chi-square-statistic-for-a-nonsymetric-table
Your problem is ill-posed.  The answer depends on the number of people the 3 were randomly selected from.
So if you are given i.i.d. observations $(X_1, Y_1), (X_2, Y_2), \\ldots, (X_n, Y_n)$, can't you just form a likilihood function $L^n(\\lambda)$ and then solve the optimization problem: $\\max_{\\lambda \\big| |\\lambda| \\leq 1} L^n(\\lambda)$? Of course, actually maximizing the above function may turn out to be computationally difficult if $L^n(\\lambda)$ is not a well-behaved concave function. But aside from that, what is the issue?
If the series looks like 1,9,1,9,1,9,1,9,5,9,1,9,1,9 ,winsoration win't help
@B_Miner The forecast using the approach I suggested would be a real number which you could then convert to a binary e.g. if forecast is less than $10,000 set = 0 ELSE set =1 . If you wish to post an actual data matrix containing nob observations on N Series AND k predictions ( length of forecast ) for each of the user-specified input/causal series . Alternatively if you don't have future values for your user-specified causals one could develop (ARIMA) forecasts for these supposedly exogenous series and then use them in the Transfer Function to obtain forecasts for the dependent series.
Formally you can do, but from practical point of view are the discrete choice variables endogenously related in your model? It would be nice to hear a complete story, and in this ad hoc case it will be more or less clear if it is relevant. The estimation algorithm probably will be slow, since you have to deal with full (limited) information maximum likelihood method estimating the parameters.
You do not have a _process_, just a random variable for a given value of time $t$. You have not specified how the potential random variables for different values of $t$ relate to each other.
And your comment about imagining that accuracy started going up... typically in one curve you want to model a single psychological process.  I doubt accuracy increasing after difficulty exceeded some point does that.  It would be tapping a different process... or just be some chance variation.
I assume you mean cross-correlation. Load both signals into matlab and use xcorr2 function
Well, that even in a group of not-significant models there is one with best AIC. Anyway, please do not use answers to extend your question.
maximum of the 2nd derivative?
Remember where these conditions come from: $y$ is the quantile of order statistic $j$ and $y+z$ is the quantile of order statistic $k$ with $k\\gt j$. Thus $y$, being a quantile, cannot be less than $0$; $y+z$, also being a quantile, cannot exceed $1$; and $z\\ge 0$. Therefore $y$ must be integrated from $0$ to $1-z$ under the assumption $0\\le z\\le 1$. When you are integrating over $y$, $z$ is a constant--and this gets us back to your question. What is confusing about the presence of a constant in an integrand?
If you are just asking for code, this question would be off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)). There is some information about generating correlated continuous variables here: [How to generate correlated random numbers](http://stats.stackexchange.com/q/38856/7290). Generating correlated binary variables is trickier; it depends on how you are thinking about the data generating process. I have a related answer on SO here: [Generate categorical variables with chosen degree of association](http://stackoverflow.com/a/20330903/1217536).
Can you expand on this answer a little? At the moment it is somewhat cryptic. Why should these be equivalent?
+1. If there are differences between the group means and it is ignored in the model fitting, then the model will fit the best approximation (within the subspace that forces the groups to be homogeneous) which still has $E(\\varepsilon)=0$, which will effectively average the coefficients over the groups. Of course, when the group means are different, this model fit isn't terribly useful, unless you're trying to make inference about a randomly selected person whose group membership you don't know.
T-tests, significance, sample distributions, and estimates of proportions deal with *conclusions* you draw from the data.  Finding "representative cases" is a matter of *selecting* cases from the dataset.  The two activities are completely different.  What, then, do you really want to do?
For starters, look for a peak in the cross-correlation. If you could put a figure showing your cross-correlation, we could probably help you better.
@rama That is the point of offset terms. There are a number of motivations for how they come about, and you might want to read about them [here](http://en.wikipedia.org/wiki/Poisson_regression#.22Exposure.22_and_offset) and [here](http://stats.stackexchange.com/a/11183/8141).
Looking at your data, both your response and explanatory variable are ordinal and there are ways you can account for both in ordinal logistic regression.  In R, the polr() function in Venables and Ripley's library(MASS) will let you do this.  Hint - the answer is "yes there is a significant relationship" - and even a Chi square test will show this but a test that takes the ordinal nature of the data into account is superior in many respects.
Really not sure how well this answers the question tbh....
And don't forget to make sure the results are CrossValidated ... seriously :)
In general, it's not possible to answer from the information given.
after re-reading you are misuding the statistical term "variance". What you might mean is to detect significant differences in the two ARIMA models which provide expectations regarding the expected value
Thanks for the tip. I added an answer that details the solution I came up with, which was in a very similar vein to what you proposed. As far as normal distribution goes, I've got about 2,925 data points at the moment. Would that be considered a sufficiently large sample size in order to apply the Central Limit Theorem?
I cannot edit it because I don't know what you're trying to say.  If you can arbitrarily add separate values to each of the numbers in a dataset, you are merely changing one set of $n$ values into a completely different set of $n$ values.  I don't see how that's relevant to the question or even to your opening paragraph.  I think anyone would grant that such changes can alter the mean and SD, but that doesn't tell us why the SD of a set of nonnegative data can be any positive multiple of its mean.
@Scortchi Ok, good to hear. But is there a risk that the high collinearity causes big changes in the parameter estimates, for instance that they reverse each other signs or something like that?
@Glen_b woah. I actually just made that one up myself. Yet another point for human's inability to generate random numbers...
This might be a better fit on math.SE since there is not much of statistics in the question.
If you use the model $Y_{t+1}=f(X_t)$ and are interested in one step ahead forecasts then it is ok. More common situation is the model $Y_t=f(X_t)$. When you divide into in-sample and out-of-sample and measure the forecasting performance you can use $X_t$ from the out-of-sample to get $\\hat{Y_t}$ and then look at MAE, MAPE, etc. But that would not be right, since in real life exercise $X_t$ are unavailable. Hence although you have out-of-sample $X_t$ you need to forecast them and use only out-of-sample $Y_t$ to measure forecasting performance.
Thanks for your input, @Macro. But how do I obtain the parametric distribution of the error of the fitted model (in R or Java/Groovy)?
+1 Finding confidence intervals on the parameter estimates is going to be a challenge in any event.
Thanks for the paper! While I find it an interesting read (I work in a biology related field, but not with gene expression data), what bothers me of the example I posted is that in that case everything follows a normal distribution, the tests are independent and yet the fdr only allow to find very few results.. Of course it satisfies the constraints, but still it doesn't have the power I would expect...
@probabilityislogic Thanks! I also welcome criticism, particularly if constructive.
thank you again. So I can use 0.0287 as SD in standard power calculation ? With example of following R code to detect 0.01 difference from control to intervention group ?

power.t.test(delta = 0.01, sd = 0.0287 , sig.level = 0.05 , 
power = 0.9 , type = "two.sample", alternative = "two.sided")

thank you more
It indeed seems to me that the intercept will be 0. I have not centered the time variable so I will probably use the model with random slope only.
Thank you for your answer!
It may not be 100% up to date, but look at the `Value` section of `?train`
`R` is widely accepted in the statistical/scientific community and many papers used it for the statistical analyses. I can't judge if the `ltm` package is reliable in all cases, but there has been a peer-reviewed [paper](http://www.jwalkonline.org/docs/Grad%20Classes/Spring%2008/modmeas/homework/ltm.pdf) about it. Just make sure to cite the `R` version as well as the package (type `citation(package="ltm")` to see how). The data size in `R` is limited by your PC's RAM and `R`s numerical accuracy is as good as the one of other packages (as far as I know).
Appropriate measures would likely depend on what you are using SVD for: it has a huge number of applications.  Could you give us some context for understanding your question?
(In addition, the distance-based clustering methods I had tried was the subgraph method I mention. I realise this is now called single-linkage clustering https://en.wikipedia.org/wiki/Single-linkage_clustering)
+1 for giving someone direction in a language they already know. Note that numpy has all of the functionality the OP has described needing.
Is this a homework problem? Could you show us some methods of solving the problem that you've attempted?
You are right to be concerned: the question is unanswerable because the standardized regression coefficients are unitless and therefore (in general) it makes no sense to compare them to the outcome variable. Consequently trying to compare their variances is equally nonsensical.
I have used the gamma distribution for these data. If you use the gamma distribution with a log link you get almost the exact same result you get from an over-dispersed poisson model.However, in most of the statistical packages I am familiar with poisson regression is simpler and much more flexible.
Your response is very insightful. I particularly appreciate the last paragraph, about the differences in evaluating the performance of ML and evaluating the performance of DM.
thanks for translating. I figured it would be an easy exercise, and I learned some R from it
I'm still somewhat new to this site .. what does it mean to make something a Community Wiki (question?) ? I understand it doesn't contribute to reputation(?) but other than that, does it matter? (I have been reading the FAQ, but some of the mechanics of this site are still a bit unclear to me). Thanks.
@Szabolcs, I actually may miss a point of your edit. But PCA (or SVD) is just a rotation itself (a special case of orthogonal rotation of the axes). Any translation (like centering) or shrinking/dilatation (like scaling) of the cloud should affect the results of this rotation.
You clearly don't want to use a straight line here. Something like a quadratic is very simple & would be the first step typically w/ data that look like yours. But that isn't really right either, because your data are cumulative, etc. The whole issue is misconceived. You may want to work with a statistical consultant.
Hmmm.  Do you know if there's an existing LMM framework (MLwin, HLM, etc.) that handles this structure?  At a quick glance, the linked document estimates independent variances ...
This point about saturated not implying a perfect fit is the most interesting part of this thread.  A natural example of such a situation would be [monotonic regression](http://en.wikipedia.org/wiki/Isotonic_regression).  Suppose, e.g., you know your values must increase over time and you do polynomial regression, *constraining the polynomials to be increasing.*  Consider data that have some error, so at some times they decrease a little.  Then no matter how many parameters you use (even when it's *more* than the number of data values), you will never fit these data perfectly.
+1 An answer par excellence. Gracious thanks !!! :)
@mpiktas It isn't as inapplicable as it looks. The situation is similar to an ANOVA after we apply a sufficiency reduction. This hints that the usual ANOVA estimates of the group means are inadmissible provided we are trying to estimate the means of more than 3 groups (which turns out the be true). I would recommend looking at proofs that the MLE is admissible for $N = 1, 2$ and seeing where they fail when trying to extend to $N = 3$ rather than just looking at proofs that Stein's estimator does what it claims to do, which is easy once you actually have the estimator in mind.
Here's an explanation and example using `R`: http://cran.r-project.org/web/packages/adegenet/vignettes/adegenet-dapc.pdf
Did you determine the three most important predictors from the data set you plan to analyze?  Or is this well known information from other sources?
In linear regression $R^2$ is equal to the squared correlation between the observed values and the fitted values - how about that?
If you were doing a confirmatory FA using SEM, I think you could deal with the missing data under the same MAR assumption with expectation-maximization/ML (e.g. http://www.palgrave-journals.com/jt/journal/v11/n1/abs/5740069a.html). However, I'm not sure how to take advantage of the all of the auxiliary information you have, or how this might apply to exploratory FA. Just a thought.
First of all, this is just a big example and doesn't really explain explain the concept of p-value and test-statistic. Second, you're just claiming that if you get fewer than 5 or more than 15 white marbles, you reject the null hypothesis. What's your distribution that you're calculating those probabilities from? This can be approximated with a normal dist. centered at 10, with a standard deviation of 3. Your rejection criteria is not nearly strict enough.
(+1) [Hoeffding's paper](http://projecteuclid.org/DPubS/Repository/1.0/Disseminate?view=body&id=pdf_1&handle=euclid.aoms/1177730150) is available online.
What are you trying to get at by using the number of days as the dependent variable? It seems (and your data seems to verify this) that people are either smokers or they are not so it could be appropriately viewed as a binary measurement. If you simply must use the number of days as the outcome, then you could use an ordinal regression model (e.g. the proportional odds model) but I'm not sure what added understanding that would give since your response distribution is basically binary.
Goodness yes; thanks for the correction.
Google Scholar finds 229 papers authored in 2008 by a Blundell. Could you give us a bit more of a clue to which one you are referring, or maybe a URI?
This type of unsupervised techniques relies on distances between observations. Do you have a natural way to define the distance between two such vectors of unequal length? If not, can you explain why exactly the vectors have unequal length?
Why not to plot it (the results in 2d or 3d main components) to get impressed yourself?
The reason $n+1$ appears in (a) is that the expectation involves $n+1$ independent variables: the $X_1, X_2, \\ldots, X_n$ from which the order statistics are derived *and* $X$ itself. Thus, the event $X\\le X_{r:n}$ describes cases where $X$ is one of the $r$ smallest values out of $n+1$ independent identically distributed values.  With this understanding of the problem you should be able to make short work of (a).
You have covariates, but you apparently only assess whether your models give the correct *total* numbers. Have you looked at whether your predictions match the actuals per case, i.e., taking covariates into account?
If I recall correctly you only need to prove that you have a Markov chain with nice properties, then the convergence proof comes from Markov chain theory. Maybe that is the reason why everybody keeps referring instead of giving exact proof.
I'm pretty new to PCA myself, but do you think it could have anything to do with the fact that there's a second component?
Q-Q stands for quantile vs quantile.  It is like log paper for normal distributions.  Things are scaled so that normally distributed data will fall close to a straight line. It provides an informal graphical check.  I think redefining the bins will help with the validity of the chi square test.
The expectation of your kernel involves third moments and thus cannot be unbiased for a parameter based on the first two moments.
I guess I don't know why you are interested in the article. They talk in the abstract about combining p-values for a combined hypothesis. If you are combining independent data set to reach strong conclusion because of the increased sample size. That is all that us meant by a meta analysis. One way to do meta analysis (also called data synthesis) is to combined p-values from independent tests of the individual data sets. Fisher's combination test is one of many that could be tried. Apparently the paper compares 6 such methods.  Meta analysis is in the list of key words in the abstract.
It might help if you explain the experiment.
Besides that fact that the former link wasn't really related to *mathematical* statistics, the latter one is neither: "Data Mining with STATISTICA Video Series" - http://www.youtube.com/user/StatSoft#g/c/B804A810436AFB03
This seems like material that is covered in many places on the site. Please read some of the existing threads. If you still have a question afterwards, come back here & edit your question to state what you have learned & what you still need to understand. Then people will be better able to provide the specific help you need without just copying & pasting their answers from elsewhere on CV.
+1. The reason "Gamma(0,0)" makes no sense is that the *double limit* of the PDFs $f(x;a,b)$ given by  $\\lim_{a\\to 0^{+}, b\\to 0^{+}} f(x;a,b)$ is undefined.  One can obtain different functions by making $b$ vary differently in terms of $a$ as $a$ is made smaller.
+1 I wanted to post something similar.
Nothing, I'm afraid.
Firstly, let me apologise for not replying earlier. I'm on vacation in Cornwall with limited internet access.
Although Bayesian and likelihood approaches are superior in my view, for now I would just ignore the $P$-value and use the 0.95 confidence interval.  The confidence interval does not need a null hypothesis to be interpretable.
I can't see why it's not idempotent: if I project the vector in the space (even if it's not orthogonal projection), and I put a constraint in the coefficients, why would a new projection of this projected vector be different from the previous one?
Hi Gung, thanks for your response. as it is written in R language and i have no experience in R. Do you think, i should try it to write in java. it is worthfull. and Second, should i take mean by stl in R to ARIMA model output?
thanks. Just what i needed.