tttjjj/elt_bench huggingface dataset:
Thanks a million @Nick Sabbe, I now realise my mistake.  I really want to be able to say that 50% of the population make up x amount of the culmative totals.  So, I think I really should be using cumsum instead... So I did this "plot(cumsum(V1))", which gives me a different graph (one that makes a bit more sense).  Thanks again for your help.
From [this slidedeck](https://www.cs.cmu.edu/~kbe/dp_tutorial.pdf), I found this [paper](http://www.cs.berkeley.edu/~jordan/papers/hierarchical-dp.pdf), the author of which also has this [introduction](http://www.stats.ox.ac.uk/~teh/teaching/npbayes/mlss2007.pdf). If those won't suffice as sources, their own references likely will. Also this [video](http://www.youtube.com/watch?v=PxgW3lOrj60) on hierarchal Dirichlet processes.
The aspect ratio must be set to 1. That solved it. Nice graph
I'm not sure why Gaussian white noise would necessarily lead to negative values. For example, $\\frac{dy}{dt} = \\alpha + W_t y$ models a constant rate of growth with noise ($\\alpha$ is a positive constant, $W_t$ is white noise). Note that the noise is scaled by the value of $y$, preventing a negative growth rate when $y=0$ and thus preventing negative concentrations. Can you provide more details about your ode?
I see you added the 0.5 but where does it come from? if you go by the definition of `M = log(X/Y)` then you can derive `X = ..., Y = ...` without a half, right?
These bands are automatically produced by `LinearModelFit`: see the help page, especially the introductory section titled "Properties of predicted values include:".
I know this is an old thread, but if you're still considering that idea, take a look at this paper which shows how to create confidence bands that give X% *simultaneous* (and not just pointwise) coverage for things like quantile-quantile curves, differences in ECDFs, etc. http://www-stat.wharton.upenn.edu/~buja/PAPERS/paper-sim.pdf
@whuber: I have updated the Question and provided the quotes where sampling distance is mentioned; also added a reference which can be downloaded. The paper "Computational measurements of the transient time and of the sampling distance that enables statistical independence in the logistic map" is mentioned under Ref[18] of the link. Thank you for your suggestions.
Thanks! That sounds like exactly what I need. I'll give it a try.
+1 for this well-balanced answer.
I’m writing a paper for a broad audience an wish to contrast those statements with the equivalent statement for the interval containing 95% of the population proportions that are likely to yield the observed number of successes. That interval is easily accessible from the cumulative beta function, Beta(x+1,n-x+1), and has exact coverage given the observed successes. It is fully frequentist, being a fiducial interval and is at the same time a Bayesian interval with a uniform prior. (Ross 2003, <www.computersinbiologyandmedicine.com/article/S0010-4825(03)00019-2/abstract>
@RoyalTS Thank you for pointing out the need for a fix to one of the code snippets. By the time I found your suggested change it had been rejected (correctly, because it wasn't quite right, but never mind that: I am grateful you noticed there was an error).  I fixed it by removing the reference to `rotation`, which was set to zero originally and therefore made no difference.
Welcome to Cross Validated. Is this a homework question? If so, you may want to have a look at [our meta-question about how to ask these properly](http://meta.stats.stackexchange.com/q/12/32036).
Automatic things are evil only if you don't know what they are doing.
What does "The marbles are assumed to be normally distributed." mean here? Does this mean the *number of marbles per sack* are assumed to be normally distributed? Are negative numbers of marbles per sack permitted (normal distribution goes $\\infty to \\infty$)?
Coefficients of a specific covariate represent measures of difference in mean response, in the same cluster, at observations for which the specific covariates differ by one unit and all the other covariates are identical. Depending on the link function, the "measure of difference" can be a difference, or a log-ratio, or a log odds-ratio. An exception is the intercept, which doesn't describe a difference, but instead gives the mean response in observations for which all covariates and the random effect(s) are zero.
I have reworded the question according to your comments. Please check if I have accidentically changed the meaning.
(+1) The dartboard-analog to demonstrate the difference between bias and variance is simply brilliant
@Chris Thanks for the link. Will try it on some data of mine.
I would check out the books by Alan Agresti. His introductory book at categorical data analysis sounds most suited to your request. www.amazon.com will guide you so I won't give the details here.
If you have such data then the regression will be possible, but otherwise your question seems too broad for anyone to know exactly what you seek. For example, there could be comments on how far this makes sense economically; how well will it work as a prediction method; whether a linear functional form is a good model.
*Why* did you want to do a regression of one coordinate on the other? The better way to do this depends on what is is you're trying to do. What's the aim? What question are you trying to answer?
@I have no way of knowing about the amount of exposure for sure, but I don't think the difference is big. What about the if assume that exposure was exactly the same for both groups.
Can you provide the code how you fit the conditional logit?
I have already read those papers and did not find them particularly helpful. What makes you think $\\mu$ is the best estimator? Why not $\\mu*$? I can assure you that there are outliers, this is why I started to doubt $\\mu$ in the first place.
Each items has lots of features, and I'm interested in analyzing some of these features to make inferences on the items in general. your "customers coming into a shop over time" is a good metaphor. so I want to learn about those customers. However, there might be differences between morning and evening customers, so I want my sample to have both, so my inferences can say whether my analyses (dependent variables) are time dependent or not. However, unlike "customers coming into a shop", I don't know the start and end periods, so I need a way to sample the whole population without knowing it
(1) "Significant" says relatively little about *amount* of the effect. Especially if your data are numerous, ANOVA can declare utterly trivial amounts of collinearity to be "significant."  (2) Why would you remove all continuous variables? What would the reason be for losing all that information?
@alto my question generally about is there some technique exist that can be trained with audio signal not separated to letters or phonemes by hand but just by providing sentence that contains in it,the same with handwriten strings or it can be analogy that we give classifer image and information what object present on image but not specify rect positions of them.
Changing the prior mean of $g$ is equivalent to changing the value of $\\mu$.
(+1): Unfortunately this seems to be an orphaned project.
@KRRG-BITS I have updated the answer with my personal opinion, although I doubt I have done any justice whatsoever and not sure how much of help it would be to you.
First, you should state what you exactly mean by linear time series. Linear can mean pretty much (linear in the parameters, linear in the applied basis functions, simple linear regression with each predictor separated, etc.)
Thank you so much for your answer...Can explain it more...It would be great,
How were the bins chosen? It's straightforward to obtain CIs for the heights when the cutpoints are chosen independently of the data, but otherwise it's much more challenging. Normally, bins for histograms are chosen *on the basis* of data characteristics, including their amount, their range, whether there are apparent outliers, and how skew they might be: this creates a lack of independence which is difficult to quantify (and therefore it ought to color our interpretation of any CIs we might attempt to calculate for the heights). You can avoid these complications by using a probability plot.
@Charlie Of course you can't match on factors you don't have information for matching.  But if they are not observed you do not know if they are imbalanced and can't directly assess their potential to bias results.
@Nick: No wonder, there's no straight forward answer to this. Thank you for pointing me to it.
It depends why you're doing the factor analysis. You say you're doing it to determine the factor structure (more strictly, a factor structure), if you have cross loadings, that's the factor structure; if you have low loadings, that's the factor structure.
I assume you mean http://wordnet.princeton.edu/?  I was not aware or this, but it might be useful.
@Mike @Henry This subtle difference, though, is crucial: binning by *sample* quantiles has legitimate theoretical justification whereas dynamic re-binning by quantiles determined by the parameters does not and suffers from several problematic behaviors.
@naught101: For long analyses, I do run it separately and save the results, usually in an `.RData` file, for input by the Sweave document.  However, there are also several great options out there for "caching" the results from a code chunk so it doesn't get rerun.
@Charlie 'below' and 'above' are functions of whatever way a reader of the page is sorting (active/oldest/votes), and in any case the order under some of the sorting criteria can change over time (including the default). As such, it's probably better to mention the person that posted the discussion you refer to, or even link to it.
+1 exactly what I thought
Increasing the standard deviation parameter is equivalent to increasing the absolute value of each realisation by exact the same amount. Thus the quantiles follow.
Hi samira, and welcome to the site! Your post seems to be a new question rather than an answer to the question. I suggest that you delete this answer and post it either as a comment. See also the [help pages](http://stats.stackexchange.com/help/how-to-answer) of this site.
See `predict` with `type=terms` for the "contribution" you're talking about, and perhaps look into component-plus-residual plots, which is a more common way to look at these contributions.  See `crPlots` in the `car` package.
I think [this post](http://stats.stackexchange.com/a/54894/21054) by whuber is worthwile reading.
Because people tend to collect data on variables that are positively correlated.
But what would those 64 data points be? It might not be good to assume a priori that the machine would run in X time.
@NickCox Thanks for your comment! Using the function `aov()` or `lm()` does not change anything as I later read the results with `coeftest()`. My question is not an issue with R programming. I am not asking for code lines. I'd like to have advice concerning how I should highlight with a graph (boxplot or other) the differences found on for the variable `pred1`. I thought about a boxplot but didn't know how I could add significant values (not a programming issue!) given that I have n-1 p.values for n groups.
Also a friend of mine pointed me to the student's T test if anyone cares to comment on that. http://en.wikipedia.org/wiki/Student%27s_t-test#Unpaired Thanks!
The best _method_ is, by definition, truncated regression. If the real question is what software can do it, then Stata's `truncreg` is one possibility. The documentation http://www.stata.com/manuals13/rtruncreg.pdf is accessible to all.
Actually, using `adf.test` in `tseries` seems simpler. I(1) is the default in both cases, as far as I understand.
Is the redundancy just in the meaning of the predictor variable or is it in the actual data such that each variable repeats the same baseline data?
Consider the outcome of rolling one six-sided die. The events "rolled a 3" and "rolled a 2" are mutually exclusive but their total probability is not 1.
Keep in mind what you are modelling. Each subject has its own intercept, and your model assumes that these intercepts follow a N(0,v) distribution. The variances reported are simply the "v" parameter here. Similarly for the time variable.
To add to the confusion, sometimes the term "$q$-value" is used to mean "$p$-values that have been adjusted by the multiple comparisons procedure". That's the meaning that I read the OP's question in light of.
Any book in this series causes so much distraction that for me it is no-go, no matter if it is math, programming or playing guitar. But, OP MMV.
According to the source code https://github.com/jeffwong/imputation/blob/master/R/kNN.R, any entries which cannot be imputed are just set to zero. The reason why you are seeing so many zeroes is because the algorithm which the package author has chosen cannot impute values for these entries. It might be better to relax the algorithm somehow to get sensible estimates for these values.
related: http://stats.stackexchange.com/questions/74627/categorical-response-variable-prediction/74657#74657
Good idea, and it basically works. I suspect that there's a tradeoff of some sort -- otherwise the `dlm` folks could simply use that solution to code `dlmForecast` -- but it does basically work.
But they would. Increased pipetting errors would increase the variance across samples as well.
Thanks for your answer. I added a new test in the question post which I reduced the effects of x2 term. I think maybe the finite sample effects are the main cause of rejecting null hypothesis. In test 2, as I reduced the related effects, it indeed accepted the null hypothesis. So what's your opinion?
@TooTone The uncertainties are variances. Thanks.
If I'm not mistaken, the PDF of the three-parameter $t$-distribution is given on page 125 of the [help file](http://cran.r-project.org/web/packages/gamlss.dist/gamlss.dist.pdf) of the `gamlss.dist` package. The PDF is also explained [here](http://en.wikipedia.org/wiki/Student's_t-distribution#Non-standardized_Student.27s_t-distribution) on wikipedia.
Re my 2nd point, I recently wrote an answer [here](http://stats.stackexchange.com/questions/12262/what-if-residuals-are-normally-distributed-but-y-is-not/33320#33320) that may help make this issue clearer. You may want to read it.
Adding to gung's comment, I would like to emphasize that not only the classification rate is important; a table with true-positives, false-positives, etc., obtained by CV, is a more complete view of the classifier's performance. Also; have you tried another classifiers? Maybe applying regularization to the logistic regression?
@whuber you are completely correct. I can only do it for sums that I know will appear.
Are you suggesting I didn't know this? :-) But because in practice we tend to get small counts out in the tails, the chi square approximation doesn't work well.  A permutation test (or, simply, bootstrapping the chi square statistic) does work: that's what I did the experimentation to confirm.
(C) When do we decide to apply evoltionary algo over convex?
See [this discussion](http://stats.stackexchange.com/questions/7308/can-the-empirical-hessian-of-an-m-estimator-be-indefinite). Short summary is that positive definitess (not negative definite think minimum of $x^2$ and its second derivative) is the required condition for solution to be a local minimum. If this condition is violated then the solution is not the local minimum.
Welcome to the site. The question has little to do with time series, so I added a few other tags. To do accurate multiple comparison analysis, you would need to divide by some sort of effective sample size, which will be lower for (positively) correlated time series. Bonferroni will likely be too conservative, so in a way what you say may be true. There likely to be time-series-specific methods of outlier detection that work around this difficulty.
Exactly. Thanks @Manuel for reminding me. I've edited there.
Perhaps see also [this question on descriptive versus inferential statistics](http://stats.stackexchange.com/questions/71962/what-is-the-difference-between-descriptive-and-inferential-statistics)
My Stata 12 install comes with more than 170 Mo of official documentation in PDF. Are you interested in developing dedicated packages (programming), applied statistics (modeling, visualization, etc.), or just tips & tricks?
There are at least two ways to bootstrap. The simplest way is to simply draw a random sample from the given observations and estimate the model b times (bootstrapping "pairs"). You can also bootstrap using the residuals from a model ("residuals" bootstrap). The first neglects the error structure in the data, which the second method implicitly assumes your model is correct. Efron & Tibshirani (1993). "An Introduction to the Bootstrap" is the place to start.
Thank you for sharing your philosophy.  I agree: it's good to help people learn to answer their own questions.  Empirical investigation ("trying"), though, is usually best when informed by theoretical considerations to indicate the limits of its applicability.  In this case you're safe with the Q-Q plot, because it is so sensitive to deviations from normality and you used a largish sample size of about a thousand, but the histogram alone can be deceiving.  (Almost *any* bell-shaped histogram "looks" normal!)
His classes are a binary discretization of income, so that does make it a classification problem (on the question of whether to model this as regression or classification, there is [a big discussion here](http://stats.stackexchange.com/questions/49953/classification-vs-regression-for-prediction-of-the-sign-of-a-continuous-respons/50399?noredirect=1#comment98886_50399)). If he chooses the threshold for discretization right, he can even solve the heavy tail issue you mention (make the class distribution fifty/fifty).
Btw, "smooth" really is a technical term, meaning that every derivative exists. `x -> e^x` is smooth even though its derivatives of all orders are everywhere non-zero :-)
A regression when all independent variables are binary is also called an "ANOVA". It is OK to use a regression analysis in this case.
A general formula will be hard to come by and impossible to write down unless you make some quantitative assumptions about how the $p_n$ vary with $n$. Did you perhaps intend that the $p_n$ be constant, as suggested in the example?
These methodology suggestions makes sense, thanks very much for your answer. I will work on implementing this in R, but as a novice I would be interested in suggestions on how to code this cleanly.
I find your waffle chart unclear, because of the excessive whitespace.
If I had to guess, in the examples you were looking at there was no residual error. In that case the line would pass through all the points.
@Glen_b: Thanks for your reply. I have modified my question. It is actually about the high throughput DNA sequencing. I think now people believe that selection probabilities in this kind of experiment follows a negative binomial distribution.
If there are two breakpoints, then a mean between the two would be a foolish thing to estimate. I would be very curious to see a scatterplot of the data.
Re 2: It sounds like an algorithm to compute the inverse CDF of a *mixture of uniform distributions* is being sought.
This pattern is expected: In figure 1, you've plottet the confidence band for the *regression line* which represents the uncertainty about the line. In figure 2, you've plottet the *prediction bands* which represents the uncertainty about the value of *a new data-point.* See also [here](http://www.graphpad.com/guides/prism/5/user-guide/prism5help.html?reg_graphing_tips_linear_regressio.htm).
Yes, you are right about that, @tomka. In a SUR model (which you can loosely speaking consider a special case of SEM models), I can get the appropriate test. Thanks for pointing me in that direction! I think I did not think about it because it seems a little bit like shooting a sparrow with a cannon, but I can indeed not think of a better way. If you write up an answer, I will mark it as correct. Otherwise, I will write it up myself soon, with a quick theoretical explanation and potentially with an example.
Yes. Thanks. Basically, I meant the test statistics of the coefficients estimated through OLS - which are functions of the estimates, the estimated standard error and the design matrix.
@Affine that's a good point. So I suppose my interest here is why the random effect didn't capture variation in the intercept. If you, or anyone has a comment at a later time, I welcome it! Thanks.
great, thanks Peter.
So you're looking for a category of books populated by only  a few, bad titles ;-)? Some thoughts: ranks will be distributed as a discrete uniform, & the number of items is presumably a Poisson.  I wonder if you could come up with a set of weights for how important each dimension is, and how much value / quality drops off with a unit decrement (this need not be constant--it could be a function), & then just do a weighted average & sort.
+1 That's actually correct, and exact, not approximate. I'd have put it in the question, myself.
Could you summarize in few words the main ideas from Perez and Baranauskas's paper, and tell us why you think that might answer the question?
Thank you so much...This is really helpful
Now that I can actually see the image attached to the question, the sorting is definitely the solution.
+1, this is a solid answer.  However, I would suggest examining a qq-plot against the theoretical gamma rather than a histogram--it will be easier to assess for deviations.
@teucer, thanks I've replied in github.
+1 for filtering out justin bieber
Posted the above comment as a separate question -- http://stats.stackexchange.com/questions/88894/different-results-with-formula-and-non-formula-for-caret-training
Would those be a series of Ljung-Box tests on the residuals, each from lag 1 up to `[To lag]`?
Thanks! It works! 
But for some zero sum rows, SAS will set some weird value instead of assigning zeros(e.g. -1.82906E-20) 
How can I get rid of it?
+ Yeah, the conditions included a humiliated nation undergoing hyperinflation, with a highly polarized government with indecisive leaders, incapable of deciding anything, essentially creating a big power vacuum.
@DavidP Maybe see it that way: If you don't use those statistical procedures, you have one more source of error. In particular, making use of statistics lets you avoid conclusions that are provably wrong (i.e. due to chance for almost certain). Even mathematical proofs often involve a lot of informal thinking and intuition. Simply put: Common sense is a major part of science.
Answering your question: Obviously not, you cannot perform 91390. It is not feasible and I don't think it could be interesting.
Instead, you use a representative proportion. In my opinion, I think that 100 would be more than enough. Why 100? Because I consider it highly enough compared with the $k$ in $k$-CV.
Assume that each sub-sample had enough realizations to be considered reliable for inference. Assume further that just one sub-sample from the A-series showed strong correlation with just one sub-sample from the B-series. To then say that "A & B have linear association" would not be valid. You would have to qualify your conclusion saying that only a sub-sample of A showed correlation with only a sub-sample of B, and try to make sense of that, and whether it is meaningful in the context of the phenomenon you are investigating.
@Michael hope this answer is useful to you?
In your question, then, you need to provide a quantitative, operational definition of your meaning of "information," because evidently it differs from the standard meaning as captured by the usual entropy calculation.
Also, check out kernel logistic regression and gaussian process classification for some probabilistic kernel-based classifiers.
What did you discover when you googled for this common question? Ask the Cross Validated community specific questions to help you decipher the *mounds* of information available on this subject.
Please add the `self-study` tag if appropriate.
Sorry, um, is covariate == feature?