tttjjj/elt_bench huggingface dataset:
Welcome to the site, @Chuck. Note that homework problems get special treatment here--we provide hints to help you work through the problem, but don't provide answers for you (see our [FAQ](http://stats.stackexchange.com/faq) & [about page](http://stats.stackexchange.com/about)). To help you better, can you say what you've tried thus far, & where you're stuck?
What is a "performance"?
So...why aren't people interested in this question?
+1 for mentioning R! =)
@Dason: Obviously it makes sense to estimate the realizations of the random effects. The whole idea is though that these values are picked at random from a zero-meaned distribution of possible batch/grouping/clusters effects. Choosing to "estimate a value" though as if the levels you see are some fixed factors of some sort is misleading.
(1) My point about the log-odds was meant more as a reductio ad absurdum than as a suggestion. (2) What's the difference between pointing to an independent variable that's been recoded to be monotonic with the dependent variable, and pointing straight at the dependent variable?
(3) what does equivalent $\\epsilon$'s mean? If some $\\epsilon$'s are equivalent, are their solutions the same?
I think you are still very confused about all the topics being discussed here and suggest you watch the first two weeks, and week 10, of the lectures here (https://class.coursera.org/ml-005/lecture/preview) This will cover exactly what you are looking to know and is probably the best reference material on the web for beginners.
One potential exception here is the interpretation of VIF does not hold for logistic regression, as there are glm weights in the variance.  The VIF is still useful but is not an actual variance inflation factor in glms.
Hi, I'm not sure I fully get the point. Do you refer to loss functions which basically count the number of correct classifications, such as the one for Manski's maximum score estimator (http://www.sciencedirect.com/science/article/pii/0304407685900090)? Proper scoring rules are different from this type of misclassification loss - they extract the full amount of information from the data, since they depend on the probabilities $\\{p\\}_{i=1}^n$ which are assigned to the $n$ outcomes of the binary dependent variable.
@Quantlbex Hmmm... I think you 're right. I deleted this part of the answer.
As it stands the question seems to be underspecified, but maybe I missed something.
A couple of additional notes: You can just list your variables in the formula as `VAR1+VAR2...`, and then include a `data=mydata` argument. This approach might be easier for you. Also, the selected columns of `mydata` & `predictdata` differ (**2**:15, & **3**:15). I suspect this is a typo.
It seems likely you do not have enough information. Travel times among successive segments will tend to be strongly correlated (usually positively) due to transient time-related effects such as rush hours and accidents. Those correlations are essential for computing correct assessments of the dispersion of total travel times.
Thank you very much ขอบคุณครับ
I think that would a senseless swop. HOMALS (aka Multiple Correspondence analysis) is - as far as I know - equivalent to usual PCA when all the variables are dichotomous.
Let's say you want to assess the mean weight of vegetarians vs. omnivores, but you know that weight will vary in part as a function of height. You don't actually care about the relationship b/t height & weight, you just want to control for that so as to get a clearer picture of how mean weight differs by diet. If you ran a straight ANOVA (in this case, equivalent to a t-test), you could not include what you knew about each individual's height; when you (statistically) control for height, you are running an ANCOVA.
Look [here](http://stats.stackexchange.com/questions/5450/what-if-interaction-wipes-out-my-direct-effects-in-regression), & at Ray's answer in particular. **There is no sense at all in worrying about the significance or otherwise of main effects if you have an interaction term in the model.**
Related question, [Does a cointegration model exist for irregularly spaced time series?](http://stats.stackexchange.com/q/2072/1036)
Uuuhh.. ? If the minimum is not unique then THE minimum does not exist. Do you mean there are several local minima ? What do you mean by "zero over a range of functions" ? Do you have an example of such a likelihood ?
@Macro: you are right, it will be a transformation towards normality only if you put it into a normal likelihood. Frankly, though, I have not seen it used otherwise. I admit my answer was incomplete, so I will edit it to make it more technically accurate.
`Each node of the tree can divide cases based on a single question only` Can classification tree, not clustering, be closer to what you want?
Logistic regression seems to be the place to start, at least. The data *are* independent, as far as I can see.
For those who want to know what waffle charts are, Robert Kosara on the Eager Eyes blog [has a piece](http://eagereyes.org/communication/Engaging-readers-with-square-pie-waffle-charts.html) about them. Take note of Jon Peltier's comments as well.
@AbhishekShivkumar Thank you :-)
If this is for some subject or for self study, please add the `self-study` tag.
First thing I do in cases like this is contact the corresponding author to see if they can reanalyze the raw data and provide you with the summary data in the way you require it for analysis.
I recognize that you are just trying to understand the analyses used in a paper, but you should be aware that stepwise model selection techniques are not valid & should not be used. If you want to better understand why, it may help to read my answer here: [algorithms for automatic model selection](http://stats.stackexchange.com/a/20856/7290).
Sorry, I am not able to put comments for this post. If this is regarded as a comment, please someone helps to convert it. Also some time series data, like tourists or airline passage data, they traditionally were modeled using seasonal ARIMA models. But as they are also count data, you can model them using GARMA with poisson process too.
@AlecosPapadopoulos Correct, I don't know $X$, nor do I know the fitted values. I have a value for the sample size but don't know if it is correct.
+1! This interpretation makes the question so much clearer. This must be what was meant.
Could you please elaborate on the settings of the cross-validation procedure (i.e., number of folds). One way to put a classifier to the test is use small number of folds. Another one is to add some noise to the test data in order to test its robustness. Also check your data, scoring 100% all times means that your problem is most probably binary separable.
Procrustes requires that points in the clouds are already matched, that is, any point _i_ in one cloud corresponds to point _i_ in another cloud. Is that your case? If not, you should somehow find such an optimal match prior procrustes, or choose another method instead of it.
I was asking why Naive Bayes supports it. So I found the answer from Michael Collins, it supports (X(i), Y(i)). [Please read (i) as superscript]. Regards, Subhabrata Banerjee.
Another possibility is to model proportion of debt collected and include debt amount as a covariate.
Monte Carlo is perhaps more appropriate once you've built a connected model that uses your sales forecasts, for example a complete supply chain model, where you're trying to understand risks associated with various scenarios or the impact of certain policy decisions (target stock holding, budget) on connected variables such as availability, customer service level, capital investment, profitability.  That becomes a tougher problem for which the sledgehammer starts to become more appropriate.
Moment matching a lognormal to this sounds on the surface like a strange idea. This is because the lognormal is *not* characterized by its moments. I will look here, but perhaps there is a way to turn the problem around a little. Let $f_0(x)$ be a "standard" ($\\mu = 0$, $\\sigma = 1$) lognormal density. For $b \\in (-1,1)$, define $f_b(x) = f_0(x) (1 + b \\sin(2\\pi\\log x))$. Then $f_b$ is a pdf and $f_0$ and $f_b$ have the same moments for *every* such $b$.
just as an update, I implemented the stochastic simulation today for a simpler AR(1) model and got the result that I wanted. I will now try to implement it in ARMA..thanks again for the answer..
-1, the comment that "too large samples increase the type 1 error" is incorrect. You may be confusing statistical significance & *practical* significance, in that a situation can exist where the true effect is not exactly 0, but so small that it is inconsequential, & we would consider the null 'true' *for practical purposes*. In this case, the null would be rejected more than (eg) 5% of the time, & more often w/ increasing N. However, strictly speaking, the null hypothesis that the true effect is exactly 0 is, by stipulation, false. Thus, these rejections aren't actually type I errors.
About your comment on TV programs in North Korea and the US... oh, never mind...
cause of the overdispersion:  This is consumer purchase data. Individual consumers are poisson, each with a rate of purchase lambda. But not every consumer has the same lambda -- that's the cause of the overdispersion. The lambda purchasing rates are considered to be distributed as gamma.  This is a common model (traces back to A.S.C. Ehrenberg), but I haven't found anything in his writing that answers this question.
I don't understand your question. The value of interest returned by `ks.boot` is `ks.boot.pvalue`, i.e., the bootstrapped p-value.
My predictors are basically responses on an ordinal scale (Poor to Excellent, 1 to 5). If my understanding is correct, linear regression cannot (and should not) be applied to such a dataset.That is why I am going with ordinal. Am I correct?
This procedure sounds extreme, and and I've never heard of anything like it. As 'learner' asks, do you know that your noise is only positive?

You might want to describe what you are trying to do with the data: your solution might involve a different method rather than throwing out a lot of data.
I disagree that that this is "well balanced".  "moderate" is not to the left of a central point in the way that "fairly great" is to the right of it.  This is fundamentally different from a likert scale.
Hi Juho, thank you for your answer. How would you actually implement that?
By the way, I fit the distribution with a Max. Lik. Est.
You may find Venables' [Exegeses on Linear models](http://www.stats.ox.ac.uk/pub/MASS3/Exegeses.pdf) relevant to the question, and perhaps somewhat provoking.
(i) I believe you could do the whole thing in a handful of lines of R code (or indeed in any number of other programs), but I don't know if that's what you mean  (in particular, one might fit a saturated model and then find the closest combination of factors to the required level); (ii) are you after all combinations near 80%, or any single combination?  (either could be done)
Hi Dikran. I should have mentioned that the performance of the weighted least squares solution was not overly good (generally 25% error at best). I don't believe the reason is that the data is easily linearly discriminable.
Hey! I'm talking about the qqplot from running plot(>lm<). Thanks for your answer. I'd like to give cs the mark, because his answer was faster. I hope that is ok for you :)
Well documented: e.g. http://en.wikipedia.org/wiki/Kernel_density_estimation and http://stat.ethz.ch/R-manual/R-patched/library/stats/html/density.html
I haven't read it yet so I won't formally answer but you can read Gaussian Processes for Machine Learning [here](http://www.gaussianprocess.org/gpml/).
"Nature of the data" generally provides only rough (or even misleading) guidance for data analysis, Raffael.  It is far more important to know the purpose of the analysis: what decisions or actions will be taken as a consequence?  Second in importance is to know something of the statistical characteristics of the data, such as their univariate and bivariate distributions.  Without any of this information available in your question, we can only provide guesses and generalities.  This makes it difficult to be useful to you or subsequent readers interested in this topic.
@chl Your right that is difficult to envision. The thinner you make the plots the more it just turns into a trend (like a time series). I don't think that would be a problem except that there is likely no logical ordering to the groups in this posters problem.
**Request**: It would be helpful for the downvoter to this answer to express their opinion/reasons in the comments. While this question is a bit more likely than most to lead to extended discussion, it is still useful to provide constructive feedback to answerers; that is one of the easiest ways to help improve the overall content of the site. Cheers.
In cases of small datasets (rows=138) and predictors and class both having nominal level variables, apart from using Naive Bayes classifier, what other methods may be used and what is the logic behind using such algorithms?
Thanks for your detailed comment, especially about terminology - that's useful for beginners to this field. I don't have a reason to assume the data is not normal distributed, I should have added that.
The model in the final histogram (which seems like the only plot that has residuals graphed) looks fine.
@Indolering Do they even do that now? Just looked at their website and it seems like they've moved on to providing analysis tools. Which makes my answer a bad one...
So, isn't standardizing the dependent variable as well as the independent variable screwing up the whole regression model? Are we still predicting Y then?
It will help if you can tell us what you are trying to accomplish ultimately beyond the specific visualization difficulties you have at present.
I think you need to clarify a great deal about this model and what the scientific question is. At this point, it's not even evident that there's a need for Bayesian methods... or methods that even call Bayesian versus Frequentist concepts into mind. Is there any reason why a simple [life table](http://en.wikipedia.org/wiki/Life_table) doesn't perform exactly everything you're aiming to do?
See [here](http://stats.stackexchange.com/a/60793/1036) for a few references of interest. I've also pointed people towards *Using Confidence Intervals for 
Graphically Based Data Interpretation* [(Masson & Loftus, 2003)](http://web.uvic.ca/psyc/masson/ML03.pdf) as well. Last time I looked on google scholar there were many similar papers in other fields as well - no economics that I remember though ;). IMO you should upload the pictures of the plots.
There are two problems with your less-than-five count rule of thumb. The first is that the correct statement refers to the *expected* counts rather than the actual counts. The second is that it is too severe. The $\\chi^2$ approximation often works well even when a small proportion of expected counts is less than five. In this case, where all the column marginals are five or less, it is obvious that *every* expected count is small and so we are advised to be cautious. Also, the correction mentioned in the Wikipedia article applies only in the one DF case; this case has 6 DF.
I may be mistaken, but neither of you have provided one example of a good retrospective study! If you feel like you do not have enough information, comment on his question, rather than answer.
@James, the `anova()` comparing `lm.1` with `lm.2` performs an F-test (http://en.wikipedia.org/wiki/F-test#Regression_problems), which basically compares the reduction in the residual sum of squares between two nested models to the residual sum of squares of the model with more terms. Therefore, it does not specifically take into consideration whether individual regression coefficients are statistically significant. Like @Manoel, I find Andrew Gelman's papers and books very helpful, especially "Data Analysis Using Regression and Hierarchical Models".
It makes sense to use a more significant baseline, maybe `now` is just too arbitrary. But I'd still be using a "age" feature, where I'd have to calculate the difference between to dates. I wonder if there is a better way, maybe label different "age" intervals, creating a new feature? But I don't know how to evaluate if this approach is better or worse than plain "ages"
Maximizing classification accuracy is NP-hard, so there are no general algorithms that can find a solution that maximizes classification accuracy, except in special cases or when the data set is small.  So for most problems, it's just not an option.
I tweaked this to make it slightly more readable & easier to find out more, I hope you don't mind.
Do you have a link between Middle Managers and Directors?  Is the DV such that it would, in real life, vary across Directors, or is it really just the rating of the DV that varies across Directors, and similarly for the IVs and Middle Managers?  In short, is there any reason to link the ratings of the 3 IVs given by a particular Middle Manager to the rating of the DV given by a particular Director?
+1 This is a very clear explanation. Concerning pgfs: you are correct that formally $s$ is merely a placeholder to track the probabilities. (Mathematically it generates the maximal ideal in a ring of [formal power series](http://en.wikipedia.org/wiki/Formal_power_series).) The beauty of generating functions is that in many cases when we *substitute* numbers for $s$, the infinite sum exists, creating a function that can be analyzed using methods of Calculus. This often produces further information about the original coefficients that would have been difficult to obtain otherwise.
If you ask participants to do a task twice, you'll want to treat the data as repeated measures.
Just be aware that if a component of your unknown distribution is a Cauchy distribution, the variance is undefined.
Please don't [cross post](http://stackoverflow.com/q/5034966/271616).  At the very minimum, you could tell people you're cross-posting.
+1 for a good answer. Regarding your comment above, note that it is possible to compute a [non-central F distribution](http://en.wikipedia.org/wiki/Noncentral_F-distribution), & from that, calculate a 95% CI (which would be a 1-sided CI in this case). It would be complicated & unintuitive, & I don't know if there's software to do it for you (there may be some obscure R package), so I doubt anyone ever does it, but it is possible.
+1 The second paragraph is great advice.  In principle, all you need is one more observation than you have parameters (including the constant).  That extra observation is needed to estimate the error variance and to obtain standard errors for the parameter estimates.  Those SEs may be large, but whether that's a problem depends on the objectives of the analysis.
Effect sizes and standard errors will answer that question very clearly. I'd suggest drawing a nice picture.
Thanks. I am trying to find a citable reference for $S(\\infty) = 0$ . Do you happen to know any?
Perhaps few here know what the mathematical level required of an odds adjuster is, or that of an odds compiler: it would probably help you to get better answers if you elaborated a little.
@Ranabir http://stats.stackexchange.com/questions/33656/show-that-for-t1-p-fracyz-leq-t-fract-1t1 (you could have posted a single question)
Indeed you are right ... but then in parameter estimation we must use MLEs? No matter the out sample judgement likelihoods  must in some way be used...
"Fitted values"?  "The models"?  Could you please tell us something about what these are, how they are fitted, and what kind of data you have?  Otherwise, it's impossible to tell what you're doing.
I worked on this problem some while ago, and used simulated annealing for longer scales, with 40-60 items. That did not work out particularly well, I have to say.
@kwak but the model specification is correct, isn't it? Btw, why the numbers of instruments should be smaller than years-1? If you look the example of **plm** vignette with EmplUK data set (p.23) they use 2:99 (I don't really understand why)?  I came across the following formula for number of instruments (T-1)*(T-2)/2+(T-3)...I'll have a look at gretl, thx!
+1 Well stated.
Your statement of Bayes' theorem $$P(\\text{model}|\\text{data})=P(\\text{data}|\\text{model})∗P(\\text{model})$$ is not correct. Your definition of the likelihood function is incorrect: you seem to be using the _log-likelihood_. Fix all that first and maybe the calculations will make more sense.
Srikant's answer is certainly a good one (and I have nothing to contribute), but you might want to wait a bit longer than 24 minutes to mark a best answer to see if anybody else has any ideas.
So, I take it you agree with me that the *unqualified* advice not to increase the sample size based on the multiple-population argument is nonsense?
Do you know anything specific about the distributional properties of your stream? For example, are they, say, positive? Bounded? Any other details you can provide will be helpful. Moments are pretty easy to calculate and store for a stream. There are also previous questions here about directly estimating *quantiles* from a stream, which sounds like what you *really* are trying to do. You might search for, and look through, those.
@Glen_b: add it as a link in the answer.
There's nothing inherent in those models that makes them causal. Causality, in respect of such models -- if it's inferred at all -- comes from other considerations than the models.
@RayKoopman Thanks. It looks like `r.test` in the R psych package computes this.  It seems like Steiger's observation about using using Fisher's Z is important. Also Hittner, May, & Silver (2003) and Wilcox & Tian (2008), compare various correlated correlation coefficient tests.
I think this would be more likely to find an answer on http://physics.stackexchange.com/
Fair enough, scatter and variation are rather the same things. Then , how does one handle the negative values to get the percentages? Ignore them (make them zero) or simply make them positive?
$\\hat\\mu=X_1$ is an unbiased estimator. The estimator $\\hat\\mu=7$, though biased, can perform better than the sample mean under some circumstances. Nevertheless the sample mean has the advantage of being the uniformly minimum-variance unbiased estimator & the maximum-likelihood estimator of the mean for a normal population, which is why alternatives are only used in practice when the normality assumption is doubted.
Thanks for your help and time. Would you mind to guide me how to make two qqnorm plots for different groups on the same plot. Thanks
Maybe this is relevant to your question: http://en.wikipedia.org/wiki/Poisson_process#Non-homogeneous
You will find links to at least four R packages for identifying multivariate outliers in the replies to a similar question at http://stats.stackexchange.com/questions/213/what-is-the-best-way-to-identify-outliers-in-multivariate-data.  That might be a good start.
+1 for mentioning L-moments! Glad i'm not their only proselytizer around here. I'm not convinced that normalizing/standardizing by dividing by a measure of location is the best approach here though, which is why I suggested dividing by a measure of dispersion instead. See point number 2 involving temperature in the question.
Check **the other tables** of the outputs and look for the coding schemes. These two modules might have coded your sex variable differently; one might have used male as reference while the other one used female.
Try ?RNG and then look at the references. R does not read from a table, it uses a variety of random number generation methods. You can also find the C code that rnorm uses [here](http://cran.r-project.org/sources.html)
even management consultants who need some statistical support
This question might be better suited to [Data Science.SE.com](http://datascience.stackexchange.com), since it's not really about statistics per se. Consider flagging it for migration if you don't get good answers here.
Linear transformations (like these) never change correlation coefficients.  The point to standardization is to improve the conditioning of the normal matrix.