tttjjj/elt_bench huggingface dataset:
Essentially I am wanting to understand average behavior based on word count, which is one of the parameters they investigate in that paper.  I want to build a Weibull distribution that is close to fitting what they've found to match web behavior based on content length, so that I can pull random numbers from it and generate a dwell time for a given page of a given amount of words.  Can I use those eCDFs in Fig. 3 to approximate lambda and k?
When you ask a question or edit a question there is a check box that appears at the bottom of the question box towards the right hand side that says 'Community wiki'. You need to check that box *if* you wish to make a question or answer CW. The idea behind CW is enable the community to edit the questions/answers collaboratively and to prevent users from gaining rep for certain type of questions. See this link for more details: http://meta.stats.stackexchange.com/questions/6/what-should-our-faq-contain/198#198
*All* random variables that take real values are "bounded in probability" according to your definition.  (I am excluding technical applications in which values of "$\\pm\\infty$" may be allowed.) This follows directly from the axioms of probability
Thank you for your precious informations, I'll look at that book for sure.
The [binomial distribution](http://en.wikipedia.org/wiki/Binomial_distribution) is the distribution of the number of successes from a known number of Bernoulli trials. Your comment that "the fact that Bernoulli trials are also described by a Binomial distribution doesn't mean that everything that is described by a Binomial distribution has to be conform with Bernoulli structure" is not correct. The binomial distribution is not appropriate for continuous proportions. In addition, I did not suggest the Gamma distribution, but the beta distribution.
So, ports 1-10, each measured for 30 days? And you want to compare the ports to each other, or just how change happens for each port, irrespective of others?
@Bogaso Please, consider accepting answers to your previous questions. This will motivate potential responders ;)
@fgnu duly noted
Consider posting this on Cross Validated.
Related: http://stats.stackexchange.com/questions/11421/what-is-the-difference-between-empirical-variance-and-variance/
The intercept is the predicted value of the dependent variable when all the independent variables are 0.

Without more information on your model, I can't say whether this is meaningful in your case.
@TWL: The topic is much too broad to discuss here. But take for instance economic evaluations of drugs: For the patient, maybe it's important to know median treatment duration, while for the health insurance company it's the mean treatment duration because they need to pay it for each patient. A curious fact: In the case of the exponential distribution, mean +/- 1 standard deviation covers 68% of all mass, mean +/- 2 sds covers about 95% of all mass. As for the normal. (But it's mere chance ;))
regarding the article you mentioned, it's an interesting observation, however it doesn't come as a surprise to me. I find a significant portion of statistical concepts and common practices as confusing and convoluted (even though I have a strong background in mathematics, and have taken a number of courses in mathematical statistics). I personally feel a lot of the concepts would have been much easier to understand if they were taught visually and using examples, instead of long and convoluted verbal explanations.
This is not a formal answer but - when a variable spans several orders of magnitude it is often easier on the eyes (and more informative) to visualize it on the log scale.
Have a look at [Chebyshev's inequality](http://en.wikipedia.org/wiki/Chebyshev's_inequality).
The usual reason for testing equality of variance this way is to run an ANOVA/regression. This shouldn't be done with the whole group vs. a subgroup, it should be done with one subgroup vs. another.
You're right that cross validation *should* involve randomly partitioning the data, but that's not what Weka's trainCV function actually does--it expects you to shuffle the data yourself first.
I don't fully understand this `contr.sum()`. I see it allows to extract the effect specific to each of the factor levels, but why isn't there a way to retrieve the average of all levels (i.e. the 'real' intercept)?
@chi MAR is exactly the sort of assumption that Manski's approach is meant to avoid. Perhaps a simple example will make it clearer. This is straight out of the paper I linked above. Suppose you want to estimate the average of some function of a variable y, E[g(y)]. Let z=1 for people with y observed, and z=0 otherwise. Also, suppose g is bounded between g0 and g1. Then you know that E[g(y)|z=1]P(z=1) + g0 P(z=0) < E[g(y)] < E[g(y)|z=1]P(z=1) + g1 P(z=0). E[g(y)|z=1] and P(z) can be estimated as the sample mean of the observed y, and the portion of the sample with y observed.
+1 it is always nice to bootstrap something. On the other hand every challenge is easy with R: http://stat.ethz.ch/R-manual/R-devel/library/boot/html/tsboot.html
Isn't what you are looking to adjust for, precisely what you want to measure - i.e. how the probability of "incident" increases with the number of patients?
Can you provide an example where they introduce a nuisance variable?
@AdheshJosh Thanks for accepting! When I answered I hadn’t enough reputation to insert the graph, now this is done.
It can be even simpler. `With[{p = Range@16/17}, N@Coefficient[Times@@(1-p+p*t),t,9]]` gives the probability of 9 successes, and `With[{p = Range@16/17}, N@CoefficientList[Times@@(1-p+p*t),t]]` gives the probabilities of 0,...,16 successes.
@Ann I do not know your data (or your theoretical model). So, it is hard to comment on this behaviour (which is not uncommon though). Regarding "running the whole model", see chapter "Covariates" on Kenny's website.
To see why averaging individual $R$ values may be wrong, suppose the direction of the slope is reversed in some of the experimental conditions.  You would average a bunch of 1's and -1's out to around 0, which wouldn't reflect the quality of *any* of the fits.  To see why averaging $R^2$ (or any fixed transformation thereof) is not right, suppose that in most experimental conditions you had only two observations, so that their $R^2$ all equal $\\pm 1$, but in one experiment you had a hundred observations with $R^2=0$.  The average $R^2$ of almost 1 would not correctly reflect the situation.
@gung I think this may actually contain an underlying statistical problem, though the OP may not realize it. ATMathew: AIC is -infinity when the fit is perfect (has no error), because there's a $\\log(s^2)$ term.
$\\mu$ & $\\sigma$ don't appear on the right-hand side of the p.d.f. formula, whereas a previously unmentioned $p$ & $q$ do. Can you clarify?
thanks for pointing out this more sophisticated technique. i'll try this and see how it compares to a more easy approach. is range defined as maximum-minimum value among all samples?
In light of these partial duplicates, please edit your question to focus on anything that is *not* already covered elsewhere on our site.
Ensemble modelling can also combine models. Look into majority voting for example. See also, stacking.
This supports an argument for increased model complexity as sample sizes become large - in the large sample case sampling error is no longer the dominant  sourse of uncertainty.  Of course this only "makes sense" in a Bayesian framework, which allows for other sources of uncertainty besides sampling error.
I did a simple Monte Carlo simulation:  `n_trials <- 10000`  `A_samples <- rbinom(n_trials, views_A, clicks_A/views_A) * rbeta(n_trials, clicks_A_past, fails_A_past)`  `B_samples <- rbinom(n_trials, views_B, clicks_B/views_B) * rbeta(n_trials, clicks_B_past, fails_B_past)`  `(mean(A_samples < B_samples))`  Is my understanding correct?
Hi @GaëlLaurans, sorry I didn't understand your comments at first. Observation numbers for sector1, sector2 and sector3 are 55, 43 and 113, respectively. Thank you for clarifying your question. I appreciate it!
No worries @Thilo
If the regression only involves one independent variable x then orthogonal regression assumes the error variance in both x and y are the same.  Otherwise for the error in variables model the direction to minimize the squared difference is in a different direction than orthogonal to the line.
I'm not sure what you are asking?
Thanks a lot guys! I was going to do a glm poisson regression and this puts my mind at ease! Also Nick, I have been taught that if the data is overdispersed then creating a model from the quasipoisson family is one method. Does this sound right?
I am not aware why u cannot describe your data using a probability model. If you mean it is difficult to fit a distribution for it, then, just try Naive Bayes Classifier NBC with a Gaussian kernel. It usually performs good and extremely fast.
For example, depending on what the figure specifically looks like, `x` and `y` could be entirely independent. Try running `plot(rnorm(20000, rep(1:2, each=10000), .2),  rlnorm(20000, 0, .8))` in R. It'll look roughly similar to your figure, even though `x` and `y` are perfectly independent.
Following [Occam's razor](http://en.wikipedia.org/wiki/Occam%27s_razor), we try to explain what we see with a simple model (e.g. few latent features).
so the above method is valid?
+1. To add just few things. 1) In ordinal regression, you can get Nagelkerke's R-square. The sq. root of it is probably the most theoretically consistent measure of association bw a nominal and an ordinal variable, like Eta is for bw a nominal and a scale variable. 2) Another, less puristic approach will be to use still Eta, but after the ordinal variable has been transformed into ranks. We often escape the problem of ordinality simply by ranking (that is, transforming to uniform distribution); this is what we do for example when we compute Spearman correlation.
Perhaps I'm missing something, but it seems like this is exactly meta-analysis (http://en.wikipedia.org/wiki/Meta-analysis). Do you have something else in mind?
Think the link is broken, did you mean [this categorical one here](http://biology.ucf.edu/~pascencio/classes/Methods/R%20demos/)? That was helpful, thanks!
The dw test is designed for checking autocorrelation in the residuals from a regression. There is no regression here, so it is not appropriate.
What distribution do you expect the numbers to follow across the groups, the uniform? You should look into this chi-squared test..
Excellent answer, thanks! Since there where no answer for some time, I also asked the question over at [math](http://math.stackexchange.com/questions/10124/), and they have some different approaches there, that might interest you.
If the grid separation is small then the point at which the likelihood ratio is maximized is close to the identified point and you will get an accurate result.  But if the grid is crude the identified point is not accurate and it would not be appropriate to use it.
+1, it's nice to see such a well researched & formulated question. Welcome to CV.
Thanks. I knew the loop was unnecessary but I wanted the logic underlying the approach to be abundantly clear. Your code would certainly produce the same results.
(1) This is a finite population situation, so a hypergeometric calculation would be more accurate (although the binomial is a decent approximation). For instance, the value for $0$ is $0.0364$ rather than $0.0440$. This has a more profound effect for larger samples. (2) Your answer seems to use "confidence" in the sense of "probability": that is not the usual meaning unless you are performing a Bayesian analysis, but you're not doing that.
It depends not on your null, but on your alternative, which you haven't specified. (In general, if you're not certain what you should be using, you should probably be using two-tailed. If you're *sure* it should be a one tailed test, you might be correct)
I don't recognize your formula from [Wikipedia's version](https://en.wikipedia.org/wiki/Akaike_information_criterion#Definition), but I wouldn't – I'm lousy at manual calculation. Don't worry about negative AICs though; see [Negative values for AICc (corrected Akaike Information Criterion)](http://stats.stackexchange.com/q/486/32036).
@M.Assad Are you using Stata 11 or 12? Then you have electronic access to the entire pdf manuals which used to be shipped as paper copies with Stata prior to version 11. 

I am assuming that you know that you can get "online help" by typing `help mgarch dcc` inside Stata. In Stata 11, the pdf documentation is a link right at the bottom of the help file. In Stata 12, there is a link to the pdf documentation in the top right hand `Also see` tab.
Absolutely the same way (corr matrix is cov matrix of standardized variables). Actually, in FA corr is used more often. However, the results, of course, will be different, if the variances of the variables were not all the same.
Being very close to fair is not the same thing as being exactly fair, which is the null hypothesis. I was pointing out one of the idiosyncrasies of hypothesis testing, namely that we often know that the null hypothesis is false, but use it anyway.  A more practical test would aim to detect whether there is evidence that the coin is significantly biased, rather than significant evidence that the coin is biased.
Reading your question literally, the answer is: ignore the subsetting of your data into chunks, and simply take the median of the entire pooled dataset. Perhaps I am misunderstanding the question. Can you not simply take the median of the set of all values in your dataset? Why does it matter that the data are in "chunks"? Is there some part of this structure that you hope to preserve?
Is this homework?  If so, please use the homework tag, and people will still try to help - but in more of a guiding rather than answering-for-you way.
@richiemorrisroe, you're welcome. I usually find if something would be convenient, R implements it this way :)
But, even if you do that, you should use a variance-stabilizing transformation, as I said in my response below. That may be an "old tecnology" from the times when generalized linear models where not readily available in software, but it is still a good approach, especially for simple models. If you have many regressors, a GLM might be better, because the parameters are easier to inyetrpret.
I agree with with Macro's answer but I think the explanation might vary slightly if you were looking at partial correlations which takes account of the interaction between covariates. From the output it looks like the correlations are unconditional and taken from the correlation matrix for the 4 IVs and the DV. The IV are highly correlated with each other except for IV4 with IV1.
Please buy a book of population genetics and coalescence. Don't try to reinvent everything by yourself...
Yeah, "performance" is probably the best term for that.
What you are asking is substantially out of my expertise, so I am putting this as a comment.  Have you looked at the Georgia Tech "Stinger" format?  http://trac.research.cc.gatech.edu/graphs/wiki/STINGER  They have some optimized tools to go with it, but I don't know if there are "R" variations.
I think your last equation should be: p= 2* P(yes|p)-0.5, (you're sign is flipped). However, this doesn't answer my question. Clearly, it's not impossible to get 9/10 to answer yes. Assume the class had 10 people. All 10 people can get tails the first time and then heads the next time yielding everyone to raise their hand. The probability of such an event is (0.5^20) > 0. Thus, I'm not quite in agreement with your last paragraph.
@JMS Good point, but Tables are useful for summarizing detailed characteristics of statistical units (cross-classified by a factor interest, e.g. clinical diagnosis or whatever), with variables of different types (continuous, nominal, and ordinal), and other results derived from statistical modeling *per se* (confusion matrix, regression coef. etc.) that won't fit into Figures (or not always if you think of Gelman's approach for showing reg. coef. as dotcharts). We need both; the question is when do we really need a Figure instead of a Table, IMO.
@MichaelChernick, then can you solve the mystery and tell us all what it is? I'm still not sure, even after looking at both of the answers here.
You may also be interested in [this recent question](http://stats.stackexchange.com/questions/10789/why-are-random-variables-defined-as-functions), which touches on some related points.
Indeed such a thing can happen - for example, in the one specification, there may be another regressor which is a common source of influence on the dependent variable and on the statistically insignificant regressor. If in an alternative specification this common source is removed, the other regressor may appear statistically significant. But in such a case you have an "omitted variables" problem -there are variables subsumed in the error term that are correlated with the regressors.
If you do not like the bias introduced by ridge regression, maybe you should use OLS.
@whuber It will be extremely close to poisson, in fact closer than a randomly generated poisson.  Run your simulation and watch the variances of each of your seven cells, and then run multiple selections of 10000 draws from a poisson(2) and watch the variances.  In cell 2 the variances will be equal, but in all the others the variance is less (I think that this is the source of your two missing degrees of freedom.)  The analysis is a bit fiddly, I'll try to add it tonight.
@fonjibe I have edited my answer to address your edit.
Can you provide more detail Justin?
@Mark.M Welcome to the site. This is not an answer, it is a comment. Answers provide relatively detailed steps to take to solve a problem.
@QuantGuy In a first approximation `diff` will do (if the discretisation is equal -- if not you need something like `diff(y)/diff(x)` or a more serious numerical method)
@juanpablo - The statement `f1g <- fitdist(x1, "gamma")` fits a gamma distribution to the original data `x1` and stores it in `f1g`.   The upper left graph in `plot(f1g)` shows a histogram for the original data `x1` as the bars, and the fitted gamma density plot from `f1g` as the continuous line.  The density plot (continuous line) is drawn over the histogram as an indication of how well the "fit" represents the data.
+1 for kernel density plots. They're a lot less 'busy' than histograms for plotting multiple populations.
I agree with @doug.numbers and am very grateful to Patrick!
Thank you. As for as I know, Gibbs sampling is to use some tractable conditional probability to generate a sequence of samples, and then use these samples to estimate some features of the distribution, such as mean and variance. In the [slides](http://people.cs.pitt.edu/~milos/courses/cs3750/lectures/class23.pdf), from page 40 to 48, they are taking about inference in Dirichlet process mixture, especially on page 42 (Gibbs sampling), I do not know how make the connection between inference with Gibbs sampling.
I want something a little more complex than this. It sounds like on average a player contributes X number of points to a game. I wanted to know if I could figure this out or a rough approximation somehow.
Correct me if I'm wrong but EM assumes a mixture of gaussian. I prefer not to make such assumptions.
After your last revision of the question, I interpret the context of the question as: Given a filtered signal C, how can I reconstruct the original signal A and the filter B, given some information about A and B? I still think that even after the revision your problem is not constrained enough. Usually in this kind of problem you should have a relatively good idea of what the filter B is (or alternatively A). Note that given any choice of C and A, you can always find a filter B such that C=B*A, meaning that without good constraints it is impossible to reconstruct the signal.
Unfortunately, this doesn't appear to be a question about [statistics](http://stats.stackexchange.com/help/on-topic) but about effective learning strategies. Perhaps more importantly, "what are your thoughts" is not a question with a clear, supportable-with-references answer, but simply invites opinion (making it off topic everywhere on the network). Either way, it's off topic here.
@Mickael: When you say "variation of data size" what do you mean? Do you mean that this solution uses more and more years of training data at each fold? Or are you talking about a trend (e.g. year 5 has significantly larger values than year 1)?
I probably shouldn't have told you your data looks discrete without actually seeing some of the numbers. Would you mind posting a summary/example of the data you're working with?
Thank you for your detailed explanations. Now I see that weighting is no panacea.
This looks like a good overview, but aren't the profile intervals based on quantiles of chi-squared distributions? Perhaps you could also clarify the various meanings of "$z_{\\alpha/2}$" appearing in your answer (as well as stating what your other variables mean, such as "$n$" and "$\\alpha$") and tell us explicitly whether $\\theta$ is univariate or a vector parameter. Concerning the first proposal: how can this deserve to be called a "confidence interval" when it explicitly requires the (true but unknown) value of $\\theta$ for its calculation? Finally, please check the last formula.
@whuber: it seems to me quite unnatural to use indicator variables for such things as days of week, but yes, you are right - it is possible to create separate indicator feature for, say, each day of week and omit splitting data. However, such algos as MD5' normally improve results, not worsen, so it may be not bad idea to split the data. Concerning stochastic optimization. I'm not statistician, but isn't stochastic optimization problem analogous to cost function minimization in regression? You just  create model and try to optimize it with respect to given data. Or am I missing something?.
It will help if you specify some background. Are the input and output required to be linear maps? Are you performing a transformation that "linearizes" a noisy input? Are you interested in variance before and after?
Alex - thanks for this, I've filed it away for future reference. The hope with this is to eventually make a predictive model and the Elkan paper seems like it might help with that, in the later stages
You may also be interested in checking out Wagonmakers et al. (2004). Assessing model mimicry using the parametric bootstrap. *Journal of Mathematical Psychology, 48*, 28-50. ([pdf](http://www.ejwagenmakers.com/2004/PBCM.pdf)).
(+1) I'd rather have a model with no discontinuities & not have to choose the breakpoint in any data-dependent way, but you've explained the issues well & that's a sensible approach.
Thanks a lot!!! See my edits above...
The model as you've written it would not be identified. If $y_{ik}$ is something like the subject $i$ in group $k$ observation then you would need repeated measurements on each subject for $\\sigma^{2}_{b}$ to be distinguishable from $\\sigma^{2}_{\\epsilon}$. Maybe there is a $t$ subscript missing from $\\epsilon_{ik}$?
The two formulas are essentially the same: 1.34 is 1.349 rounded down and 0.9 has been increased to 1.06.  But that change makes little difference in the appearance of smoothness, so we might just as well recognize what's going on and provide a simpler formula: **divide the smaller of two estimates of the standard deviation by the fifth root of the number of observations.**  The two estimates are the sample standard deviation, *s*, and a rescaled IQR (which agrees with *s* asymptotically for normal distributions).
To answer "more likely" we would presumably have to posit something about the joint distribution, not just the margins.
+1. Care should be taken with these kinds of things. For example, if you fit a random effects model with 2 random effects in `lme4`, the covariance between the random effects is automatically estimated (you have to specify them as independent to disable this). Therefore, if you delete one of the random effects from the model, you've actually deleted **two** parameters, so any inference should make note of this.