tttjjj/elt_bench huggingface dataset:
@Brani (1) R remove statistical units with missing values before estimating the effects; compare the above results (especially the DF for the residuals) with `summary(lm1 <- lm(y~A+B, data=df[-sample(1:nrow(df), 10),]))`. Of course, it is important to know whether data are missing at random or not, but see [Treatment of Missing Data](http://bit.ly/gISQy8) by Howell, for instance. (2) The `effects` package is useful to display predictions for complex models (e.g., incl. interaction). Here, you can try `plot(effect(term="A", mod=lm1))`, or browse the on-line, `help(effect)` for examples of use.
This works well and seems to make sense for my data set.
Welcome to CV. I don't recognize the upshot of your question. Why couldn't you just use standard regression?
Still could not get a better convergence by adapting IRLS. :'(
Je suis française.. thank a lot, it is the first explanation that I can understand with my poor english !
Can you be a little more specific about what you are trying to do with the algorithm? A Monte Carlo algorithm can mean a lot of different things.
Ah, this confirms my fears. Thanks for your help Michael and whuber!
I agree. Also, depending on the domain a weighted composite of categories might be a more appropriate index of an overall rating.
@chl. absolutely
Yes, this is true, but for general regression problems (as opposed to anova or designed problems), X really isn't fixed but are observations from the underlying process. However, for the Poisson case, the point still holds, since mixtures of Poissons aren't necessarily Poisson.
@Anna Varvak - as far as I understand, you do not need an endorsement, check [link](http://arxiv.org/help/submit). You just need to register.
@jkndrkn (My apologies for not @-notifying you before, I hadn't realized you weren't the OP until just now. ) ...
Yes, you can quantify the bias under a given set of assumptions via simulation, for example. But why would you choose to do something biased instead of actually using a proper sequential analysis in the first place?
To be clear; are you fitting the same model to every subset?
In many cases, intervals (which show effect sizes) are more meaningful that hypothesis tests.
As you tagged this question with `factor-analysis`, could you expand a little bit more on the purpose/context of your study? Especially, your statistical units are considered 'outliers' with respect to what?
@thebigdog, what else do you need than the `dput`?
Welcome to the site, @hussam. Would you mind expanding on this a little w/ a few sentences to explain how these options deal w/ both problems?
+1 for the reference.
This paper : http://link.springer.com/article/10.1023/A%3A1010920819831#page-1 has a very similar approach, but considers the case where the output classes are not ordered and thus gives a different expression.
Try `Rattle` package in R.
So let's say if I analyzed a non-linear model and generated y_observed at time = i. Can I do a paired t- test that compares each observed to actual value at time i? Let's also assume that I have the observed data for 100 time intervals, and forecast my numbers to the same intervals
Then there are a number of choices: Gain scores, ANCOVA or possibly mixed models.
@prob I question why it makes sense *as a generic procedure* to average over all correlations.  In effect this procedure substitutes the definite and possibly quite wrong answer "zero!" for the correct answer "the data don't tell us."  That difference can be important for decision making.
Found the answer. The cobs package in R will allow for a constrained spline solution. As per http://stats.stackexchange.com/questions/29400/spline-fitting-in-r-how-to-force-passing-two-data-points?rq=1
Please do not [cross-post](https://stackoverflow.com/questions/24301743/which-machine-learning-algorithm-is-the-slowest-but-surest?noredirect=1#comment37556042_24301743). Decide which SE site is best suited for your question.
Thank you very much! I just checked loglinear analysis and chi square and realised that I can't use them as they demand different participants for each condition! And I am again completely confused! I thought of repeated measures logistic regression but I can't find more details! Any idea? Thank you again!
(1) which parameterization of which specific 'skew-normal'? (I've seen more than one thing get called that) (2) when you say "the formulaic parameter estimates" you imply (a) a closed form exists and (b) that there's only one --- yet you mention both ML and MoM, which won't generally be the same (& the ML estimators in particular might not be closed form). More information is required!
'direction' is simply direction on the real number line ... i.e. the sign. It's not about outliers, at least not directly; the central issue lies elsewhere. The shape of the distribution of the residuals isn't especially important in telling you about good fit. It's the *size* (and in some cases sign) of the residuals.
Hadley Wickham's product plots seems to a fruitful approach to what you are suggesting, see [this answer](http://stats.stackexchange.com/a/20900/1036) with an example and further reference.
*A would like to swap with B* is the key.  To take one of the common Monty Hall explanations:  Imagine there are 1000 prisoners:  A asks the jailer who gives him 998 names.  Clearly we just learned a lot about the one guy who is *not A* and who is *not named*.  But we didn't learn anything about *A*.
Are you familiar with the beta distribution? It's only defined on the [0,1] interval and its mode can be shifted with alternative parameter specifications.
This is a very common insurance problem, combining modelling of the rare event with something like a gamma distribution for the impact - suggest you look in the actuarial literature.
Sure, you can. But the question is: is it the sensible thing to do? Start by telling us what exactly you are trying to model. Do you want to know whether the rule depends on the node number? Do you want to be able to predict node2 based on node1? Or what? In summary, WHY do you want to use regression?
The answer of 25000, which is obviously ridiculous no matter what the true state of affairs may be, indicates that a power calculation is not appropriate here.  The answer of 9228 I obtained guarantees that no matter what the distribution may be (provided only that its values all lie between 1 and 100), 95% of all simple random samples (without replacement) with n=9228 will lie within 2% of the mean.  Power is not relevant!
*Why* do you want to apply a transformation?  What analysis do you intend to do subsequently?
120 values is not small. This is intersting to me. Since I don't live in an R world, I can't duplicate the 100 values and thus explain this conumdrum. Please attach the 100 values to your post and I will put this under my microscope.
This is a great & very authoritative contribution, thanks. I hope we'll see more like it in the future.
Tell me if this doesn't make sense (it probably doesn't). If a PCA biplot depicts ~ Euclidean distances between the samples with respect to the goal of PCA (max(var(X)), then wouldn't the same be true of PLS, but with respect to max(cov(y, X))?    The problem I see with respect to applying a transformation (like Hellinger) is if you only apply it to one of the two matrices.
Thank a lot! To be more MATLAB specific, function `chi2cdf` calles `gamcdf`, which calles `gammainc`. In `gamcdf` I've changed the line `p = gammainc(z, a);` to `p = gammainc(z, a, 'upper');`, saved it under another name and changed the call in `chi2cdf`. Works great!
I could not reproduce this error with dataset cars and fitting dist~speed. It would be way easier if you could provide a runnable example (with actual data, maybe a subset is enough). Without seeing the full statement assigning a value to xxx, one has to suspect that the error occurs there (maybe a typo)
@B_Miner I meant interval in the signal processing sense and have edited above. Std dev is effectively the mean or expected variation.
Peter, Meta is for questions *about this site.* I suspect you may have been thinking about "Community Wiki" status, which is for useful, interesting questions unlikely to have an objectively best answer (or which are likely to need collaborative efforts to answer at all).  Accordingly, I have interpreted your suggestion as a CW request and implemented that.
What the class balance between drop-out and return? With what metric and cross-validation strategy are you evaluating your model? Are these predictors models? What subset of the data are they trained on? Have you tried a nonlinear model?
@kwak: No idea, but if there's no implementation yet, it shouldn't be too hard to translate the MATLAB code to R (I can try to whip one up if there really isn't any); the only prerequisite is a decent generator for pseudorandom normal variates, which I'm sure R has.
Hi Uri, these 2 studies are the results of 2 different centers which perform the same surgical procedure to correct the same illness. One group has basically excellent results, the other one very bad, so the confounding variable is difficult to find. I guess a variable would be that one group has much more experience than the other to perform this kind of surgical procedure....
Thanks. I'll take a look at the paper you provided.
Welcome to the site, @Neo. If this question was motivated by a class / textbook exercise, please add the `[self-study]` tag.
Is test time 1 for X the same as test time 1 for Y (i.e. are these paired?)
@xuinkrbin. its a simplification from the more fundamental form of it: $10\\cdot\\frac{\\sum\\limits_{j=1}^{n_i}r_{ij}\\space +\\space 10}{10n_i + 20}$. The bottom gives the maximum possible score achievable from N+2 ratings, while the top gives the sum of actual ratings, plus one perfect score. This is in direct analogy to the one dervied for the binomial estimator.
:No it has to be Q~$Gamma (n,1/2)$. But should **n** now be n/2.Then this has a Chi Squared distribution with n degrees of freedom
I think I get the gist, but the specific question is unclear. Are you asking for the chance that this process reaches $4$ *at least once* in $10$ steps?  The chance that this process *ends up* at $4$ after $10$ steps?  Something else?
+1 from a fellow visual learner :)
2) fixing the expected slope to be equal to the maximum value of slope in the population and compte the p-values of all population.
.do you have link for this handbook?
@Fojtasek haha, yeah. I just figured it out and answered my own question. I think the actual act of writing down step by step all the issues I was having is what allowed me to realize the answer XD
Each principal component is a linear function of the variables. Identify the coefficients of the variables in each component. The most influential is not necessarily the ones with the largest coefficients because they have different scales but the mean times the coefficient will give you an idea.
@Christoph_J I am not sure I understand your question. Did you mean to write $y_{t}$ above? You should also tell us more about the actual problem you are dealing with. My temperature example was just a toy model to highlight the issues. There are several solutions to deal with AR, the easiest of which is the Koyck distributed lag specification, which boils down to estimating simple equation with an $MA(1)$ error term. However, you should still conduct some sort of autocorrelation test, like the Durbin-Watson, though that can give you a false positive if you don't get the specification right.
The best test probably depends on why exactly you're testing for normality/exponential (so some background would be helpful) but you can always use the Kolmogorov Smirnov test to test whether a given data set fits any pre-specified distribution (http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test). There are lots of methods used for the normal distribution specifically: http://en.wikipedia.org/wiki/Normality_test
@MarkSAlen That would be Stevens, S.S. (1946). On the Theory of Scales of Measurement, Science, 103 (2684), June 7, 1946, 677-680.
@TheGrapeBeyond: the loadings (aka rotation matrix) are the eigenvectors $P$. The eigenvalues $\\Sigma$ give the variance covered by that component.
how can I get vec command in R      thanks
And someone downvoted my answer, too--but that's beside the point. Thank you for explaining your interpretation: finally, it is possible to assess your answer on its merits. Re "require analysis": In general, the answers that are appreciated on Stack Exchange *do* offer analysis and justification in addition to opinions and recommendations. You can *expect* to receive downvotes for answers that do not have such support--and, as my experience in this thread shows, sometimes demonstrably correct answers *with* such support get (silent) downvotes too, however unfair that might be.
@sth the outputed confusion matrix should be the full unbalanced sample size.  The sampsize argument is used on a "per-tree" basis.  I admit that the outputed confusion matrix is particularly confusing.  The sampsize argument says for each tree in the forest sample 70 of class 1 and 70 of class 2 (with replacement by default) and then split this data into a training and testing set.  So each tree has a training and testing set.  Can you elaborate on anything else that is specifically wrong besides the full confusion matrix.  I would potentially open a new question so I can post code as well.
This was the original (1908) *definition* of $t$: everything else known about it comes from this characterization.  The Wikipedia article on the Student t distribution provides a reference to [Fisher's 1925 paper](http://www.sothis.ro/user/content/4ef6e90670749a86-student_distribution_1925.pdf).
I recommend this simple and [concise paper](http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.diva-portal.org%2Fsmash%2Fget%2Fdiva2%3A288565%2FFULLTEXT01.pdf&ei=nFQUU66qF7OM7Aanh4DoDA&usg=AFQjCNHvHTvq7t29fv9cIVW_0qLsLWNzig&sig2=pGvoqXagrz4Af3ExKCDaTQ&bvm=bv.61965928,d.ZGU): 
 M. Borga, T. Landelius, H. Knutsson (1998). A Unified Approach to PCA, PLS, MLR and CCA.
The confidence interval of an estimate is not directly related to mean or median. Your question doesn't give enough information to give solid advice.
@gung - I've addressed the issue with R; there is a small, but critical difference in how multinom and mlogit interpret formula coding for the basic regression. The major question was regarding the modeling framework and that seems to be okay. Multicollinearity was/remains a concern. Thanks for the suggestions.
How much milk do you drink with your bread?
"Statistics inferes from data; Informatics operates on data." This is really what I want to confirm. For the inference, always based on the probability distribution, it should include confidence interval or hypothesis testing. Otherwise, you are just operating on data.
The formula for the variance of $X+Y$ is wrong.  It should be $$\\mathbb V(X+Y) = \\mathbb V(X) + \\mathbb V(Y) + 2\\text{COV}(X, Y).$$
+1. I did not know that distribution had a name, thank you.
Thank you for answering--that information is quite clear. But what is your explanation for why the BP/CW tests appear (strongly) to contradict the White test, which would also satisfy your description?
@Aksakal yes, each measurement has a different uncertainty
It seems an eminently reasonable test statistic, since it's completely interpretable and readily understood. Indeed none of the possible statistics will have a symmetric distribution. Let's forget the specifics of the Fisher test for a moment - if that statistic is meaningful for you, you can calculate an [exact test](http://en.wikipedia.org/wiki/Exact_test) on that basis (using hypergeometric calculations to find the probabilities). If you want to show that they're inducing the same ordering in all cases, that's probably a new question.
I think you might want to rephrase your question. As stated it seems like your focus is on possible non-linear trends in the data and how linear models may be modified to facilitate these. But your real issue is lack of data and how to avoid overfitting, while at the same time modeling these weak predictors. To my knowledge SVMs do have built in regularization so may be useful, but so would any model with regularization.
There is some confusion in this response! For instance, it says " Now in Bayesian statistics there is a sufficient statistic for the parameters of the Cauchy distribution and if you use a uniform prior then it is also unbiased. ". It is difficult to make any sense of this! First, the Frequentist and Bayesian concepts of sufficiency are very close (and I believe can differ only in some strange, infinite-dim sample spaces, so for the real line are the same). There is no sufficient statistic for the Cauchy model, of fixed dimension!, simply (the complete data obviously are sufficient).
@Askoli: No, the Kolmogorov distribution does *not* have support on $(0,1)$. Its support is unbounded.
Andre thanks for the hint. Unfortunateldy I am new and post it initially in Stackoverflow where I was advised to post it in CrossValidated
The question is about the mathematical statement of $H_0$, I think (the mathematical definition of "association" here)
Based on your explanation, I would think Random effects would be more appropriate for my data-data are highly correlated and clusters are very large in number. However, it fails the Hausman or Mundlak test for heterogeneity, which suggests that RE would be inconsistent. So I'm torn between which one to use.
May be start with 2, then n. What is the sum of two independent poisson random variables?
For questions which fall into the category of homework, "the community's policy is to provide 'helpful hints'" (hover your mouse over the tag for a few seconds). While this is a good answer, I think it goes a little beyond the 'helpful hint'.
Just for fun, I did a brute-force calculation in Mathematica.  Its answer is a ratio of a 21,574 digit integer to a 21,571 digit integer; as a decimal it's comfortably close to 900/e as expected (but, since we're asked not to post a solution, I'll not give any more details).
(1) In how many dimensions are you working? (2) "Popular" is not very clear. Could you replace that with a criterion that is relevant to the problem you face?
Agreed, solid data and metadata first.
@Michael The peaks are locations that do not exceed a moving maximum; this makes them fast and easy to compute: there's no numerical search, just a simple $O(n)$ scan. The advantage of using a differentiable smooth is that it can interpolate peaks between given x-values: this is useful for coarse or uneven x-resolutions.
Overfitting occurs when your model has a lot of parameters relative to the number of observations and starts to fit the noise as well as the signal. That can happen even when you're not estimating using the data you want to predict. Information criteria such as the Akaike and Schwarz are useful for pruning your general model back to something appropriately parsimonious.
@Pulse, The default is as I described,  the baseline level will be 0 in all dummies and there will be one dummy variable for each of the other levels.  The corresponding variable will be one for that level and 0 otherwise.  This lets you compare each level to the baseline level.  There are other options for creating the dummy variables and the interpretation will depend on which one is used (but you can still look at which terms will be what and see the comparisons).
I am going to word my opinions on this as an answer
I am not sure I follow your meaning here. You fit the model to each $K-1$ subset and then test on the remaining fold. Iterate through all $K$ folds and then resample as much as you like. I am not sure how that fits with this boosting algorithm, it seems like you are leaving out most of the details of the boosting procedure.
I appreciate any ideas,thoughts,suggestions,...
Dear Gung & Placidia, thank your interpretations. Sadly I think I did not write my question clear enough. I wanted to know "how do I know the baseline value to which LandUseLow is compared to?". The value of LandUseHigh is not shown in the table. Of course I can look up by hand, but how would I report this in a paper so that the reader can follow? Thank you VERY much for the hint on the interaction! I did not think about that before (+1).
Here's something interesting I found about divergences recently -- each step of Belief Propagation can be viewed as a Bregman Projection http://www.ece.drexel.edu/walsh/Walsh_TIT_10.pdf
Regarding (**3**), to avoid a computation, you can employ the following argument: Let $d$ be even and such that $d = 2 k$. Note that $Z_i = X_{2i-1}^2 + X_{2i}^2$ is an $\\mathrm{Exp}(1/2)$ random variable. So $\\|X\\|^2 = \\sum_{i=1}^k Z_i$. But, then $\\|X\\|^2$ is just the time until the $k$th renewal of a Poisson process with rate 1/2. So $\\mathbb P(\\|X\\|^2 < 1 ) = \\mathbb P( N_{1/2}(0,1) \\geq k) = e^{-1/2} \\sum_{x=k}^\\infty 2^{-x}/x!$. The tail of the Poisson is dominated by the leading term, so $\\mathbb P(\\|X\\|^2 < 1) \\sim e^{-1/2} 2^{-k} / \\Gamma(k+1)$ as $d\\to\\infty$ (Again: $k = d/2$).
@PeterRabbit, Can you define a clear cut between future and past? e.g. For your samples, $y_i$, you know when $i > t$, the noise source $R_2$ will play a role.
Would you mind developing your reply a little bit? Are dummy variables intended to code for missing values? How would they be used for plotting purpose?
Cyan, many thanks, I got very busy, I hope you are still fresh on the subject.I have the following questions: 1- Is Your derivation of the posterior based on your initial assumption in splitting Y into even and odd samples? If yes, I cannot understand the relation. 2- As far as I understood the posterior your recommended is inverse-wishart. However, sampling inverse-wishart distribution even with your recommended priors does not guarantee that non-tridiagonal elements of covariance remain zero. Have I missed any points you have been trying to make?
I know that `genlinmixed` only reluctantly gives tables of parameter estimates and their standard errors, but you could possibly get that information for us?  It is almost impossible to answer as is, I think.  If you are at all familiar with other statistical packages (e.g., R, SAS, Stata) I would encourage you to try one of those.  The syntax for this type of model is much easier, and you get standard output that can be readily shared rather than "a red line instead of a gold one".
Any suggestion of how to initialize the `lbfgs_malloced` array?
Also, I should make a note here that I was attempting to "simplify" the categories by calling them Healthy, Sick, Dead, Diapause.  The truth is this is a parasitoid study, and I was concerned my outcomes might just confuse people.  The actual categories are "Para with hatch", "Para without hatch", "Healthy host, no para", "Diseased host, no para".
Even though the slopes that I calculated from the predicted values fit nicely to that table, im not shore if this is the right interpretation of the table above, as it is not very intuitive. I hope I didnt confuse you too much. Don't know how to put it more precisely :/
+1, That is the most clear explanation I've ever seen.
You could look at the code that's used to calculate it, by looking, for example, at the source of the polycor package in R - http://rss.acs.unt.edu/Rdoc/library/polycor/R/ or similar in Stata.   I would be surprised if it were straightforward.
(+1) Nice response. The first paragraph is very well put.
+1 for example with trace.
Also, it's a good argument for restating questions along the lines of *"on a scale of 0-10, where 10 is absolute agreement, and 0 is absolute disagreement, what do you think of the statement: ..."*
If this is for homework (or self-study), you should add the `self-study` tag.
That is probably because they are not so impressive as others :) http://i.imm.io/rSJo.png The values here are randomized a little to show the number of objects (you can use different line width instead). Even with this toy data you can see some strange clusters and properties (only four objects have features 3 and 4 equal one). Changing the order of the features will give you more interesting results.