tttjjj/elt_bench huggingface dataset:
Part V of BDA3 is titled "Nonlinear and Nonparametric Models".
@Scortchi: (+1) to your comment. Years ago, I remember working out that it is indeed impossible in the three-group case unless the level of the test is very small (something below $0.005$, if I recall). It comes down to relationships on the ratio of cumulative $F$-distributions with different numerator degrees of freedom, evaluated at a fixed level and fixed denominator degrees of freedom.
+1 for heteroskedasticity, which I should have thought of myself.
OK, so chi-squared is asymmetric.  Why, then, compute the *right* tail area and not the left?
This is great help!! Thank you alot.
"General Linear Least Squares" is a non-stochastic description of Ordinary Least Squares (OLS) in a standard statistical multiple regression model. The critical _linearity_ characteristic in OLS has always been about the _unknown coefficients_, not about the regressors (regressors that are included in the design matrix). The clarification question remains: is both $x$ and $y$ in your polynomial considered as "independent/explanatory variables", "regressors"? Does another variable $z$ exists, non-identically zero, for which you want to fit $z = \\sum_{n=0}^{N}\\sum_{m=0}^{N}\\beta_{nm}x^ny^m$?
A real life example for the data would be temperature during seasons (almost steadily increasing during spring/summer and almost steadily decreasing during fall/winter). The opposite example might be stock market data or just random data. Maybe there is also a standard statistics function available which I don't know of.
Thanks to @chl.
It's not homework :d.  i don't understand your response, are these functions of a statistical computer program? I thought it had to be solved by transforming it to a normal distribution, but that did'nt work out well. Thanks for the reply
(note its setting range to 1, not variance). duplicate of http://stats.stackexchange.com/questions/41704/how-and-why-do-normalization-and-feature-scaling-work?rq=1
@user2534: if you're going to do any serious stats, you should get a stats packages, such as R, SAS, Matlab, Stata, etc. etc.
This can happen, especially when your dependent variable has a landslide on either 1 or 0. If you already have 95% of 1's in the variable, then just by guess all of them as 1 will get you 94.7% accuracy. The predictor may have served to up the % slightly, and yet it may not have predicted enough of the variability to be a statistically significant predictor.
Hi Tom, I can't get the article at the moment, but I believe that the proposed transformation would be for consecutive order statistics (the 'gaps') (in which case your expression is missing a set of parentheses), and also yields an exponential. It should work just fine. Indeed, it may be slightly better (in that it would possibly give better power against more interesting alternatives)
@StasK, the regression syntax for SPSS *does not* ask for stepwise regression. It is an unfortunate side effect of code produced from menu's, it always includes the `CRITERIA` statement, even if all variables are included in the equation. The relevant portion of the SPSS code is the `/METHOD=ENTER` portion, it would say `METHOD = BACKWARD` or something like that for the model selection techniques. I am going to roll back, andrea's solution was correct previously.
The limits for the marginal of $X_2$ are not from 0 to 1 except when $X_2 = 0$.
Big thumbs up on the graphs.  I think they help ground it very well.
I spent 2 hours yesterday phrasing this question and trying ways to create a sample data set. Can anybody help?
@Quantlbex It's so detailed, in fact, that it's wrong.  (The error occurs in replacing $\\sqrt{b^2}$ with $b$.)
For this to be an answer, you'd have to explain how this works. It's not immediately clear that it does.
The kind of setting that I testing data splitting on was n=17000 with a fraction of 0.3 having an event, and having about 50 parameters examined or fitted in a binary logistic model.  I used a random 1:1 split. The validated ROC area in the test sample changed substantively when I re-split the data and started over.  Look under Studies of Methods Used in the Text in http://biostat.mc.vanderbilt.edu/rms for simulation studies and related papers giving more information.
It's worth noting what happens in the limit as $\\lambda$ goes to 0 if $X$ has full row rank or full column rank.  If $X$ has full column rank, then in the limit, you get the pseudoinverse $(X^{T}X)^{-1}X^{T}$.  Similarly, if $X$ has full row rank, then in the limit you get the pseudoinverse $X^{T}(XX^{T})^{-1}$.  So, this works out as we'd expect.
Interesting, Fisher's test crashed on R.
thanks mpiktas. chl suggestion is also a good direction on how to handle such data.
@suncoolsu I edited the question on Mengzhen's behalf to attempt to make it clearer
thank you for the explanation and proper terminology / reference. The audience is readers of a scientific journal and the topic is variance decomposition; understanding the concept of a log transform is a pre-requisite but I still wasn't sure if this presentation required further justification - roots are a good alternative. Thanks.
Gotcha - just wanted to make sure derivative was not something besides a difference or % change.
I wanted to say: dimensionality reduction won't help computationally (and thus isn't necessary). 100 variables isn't much. It is 50K cases which is much. But as you've just said, you don't want to lessen their number and you insist on complete 50K x 50K matrix. I don't know how to help you.
@dlaser.  It sure does.  It's got a great API-- rather than building something yourself, you should submit patches to caret!
If i'm not mistaken, this would be an appropriate [guide](http://rtutorialseries.blogspot.com/2010/01/r-tutorial-series-regression-with.html). Also, because I have binomial predictors, I would simply use 1 and 0 for them, instead of the continuous data the tutorial has.
"measure relative snow depth" isn't a hypothesis testing problem, but an estimation problem
The amount of factors you can reliably include in models depends on the sample size, which you omitted. Secondly, I don't see why your results should follow the trend of theirs considering you are working in an entirely different context.
See my comment to the referenced question.  For the theoretical question, read about Brownian motion and random walks.
*Random* means unknowable, except in terms of distribution. *Correlated* means (by certain definitions) a quantified measure of association; and one way of looking at association is that knowing something about *X* will tell you something about *Y*. But is noise is synonymous with *random*, then knowing about some *X* will not tell you anything (no association). Colored noise (if you mean the stuff that psycho-acoutistics people and engineers are talking about—pink noise, etc.) is likewise uncorrelated. The colored noises would simply have different distributions than Gaussian noise.
This question has several aspects: (1) show that $(X,X+Y)$ is Normal, (2) compute the means, and (3) compute the variance-covariance matrix.  For which of these do you need guidance?
(1) I believe you want to use `Constant`. (2) I haven't seen MSE being used to compare estimated and true coefficients, only estimated and true sample values; maybe I am wrong... In any case if you have 3 coefficient that you estimate why would you divide by 4? :D Finally check the functionality of `lagmatrix` in conjunction with `fitlm`, it will allow you to get OLS estimates for what you describe.
Add group identity in the regression and see if that is significant. Also perform another regression of narcissism on group identity to see if there is an indirect effect from group identity on narcissism which in turn affects aggression.
This question appears to be off-topic because it is about how to use r.
BabakP, the outcome is a 0-1 variable: they person was either sick or at work.
The power spectral density of a white noise process (random process) has constant value for all frequencies. Someone thought that this was a property of white light (equal energy at all frequencies (in a certain range of frequencies)) and named the random process white noise by analogy. People liked the name and it stuck. In fact, white light has equal energy at all _wavelengths_, not frequencies, but it is too inconvenient to correct the name now.
I get the same fit and the same coefficient for Proficiency, but upon closer inspection the values for the (Intercept) and L1 (i.e. language groups) change dramatically.  (Everything else appears to stay the same.)  How should I interpret this? See [illustration](http://tinyurl.com/d8twfcp) with the same model fitted on 2 data sets: dat3(English proficiency = 1) vs dat (English proficiency = 0)
This would be the approach when the CDF is only approximated empirically. It gives lousy estimates of the PDF, though.
I forgot to add that you should compare linear mixed models fitted using ML and not ReML. (`REML=F` in R) You appear to use the default option (`ReML=T`).
Hi jbowman, I just want to say waw, thanks a million. I like the way you reformulated the problem, it is more handy. I suspected from the beginning that my problem basically was in the formulation.
@nicksabbe since the 2 symptoms are **independent**, we know that $P(S_1 and S_2)=P(S_1)P(S_2)$ (the probability of having both symptoms in the general population [with and without disease] is 0.005 x 0.008). And also I know that $P(S_1, S_2|D) = P(S_1|D)P(S_2|D)$ (the probability of having both symptoms in the diseased population is 0.3 x 0.25). And this does not add more information to my problem (*I think*)...
OOB is ambiguous here: this is about out-of-bag errors, not out-of-bootstrap (which would be for the non-aggregated models).
thank you for the suggestion and the very interesting article!
This is certainly off-topic for this site. I'll look into whether it fits Cross Validated.
@user974514 - thanks, I changed the col header to func based on the method you provided and it worked :)
@whuber Aviv has categorized the type of questions asked of the students not the answers. The question he poses is whether the types of questions asked after learning have the "same" distrbution as the types before learning> This is a simple comparison of two frrequency distributions to see if they are nearly the same. The chi square or the exact test for the contingency table is exactly the required test for this.
just curious why the title was edited with lowercase letters?                              Also, any input from someone out there would be greatly appreciated. Thanks!
You mention two sampling times as repeated measures, but your description of the variables does not seem to include it.
great answer... deserves more reads and upvotes
+1 for Orange. They have great documentation that shows how to get up to speed quickly, and you can use either their visual-programming GUI or call libraries from Python.
Have you considered vector-valued autoregressive processes as a model?  http://en.wikipedia.org/wiki/Autoregressive_model .
could you pls paste the output of dput(your_data) ?
I want to do some predictions, the set of my input data is normalized [0,1], so for the output I've used a sigmoid function. for the training I've used the mean squared error, and the weights are randomly chosen between 0 and 1.
Nice explanation. This seems to be a common confusion when using R.
Can you say something about where this question is coming from? It doesn't make much sense to me. It looks like a homework problem, but it is confused, so (hopefully) no one creating &/or assigning HW problems would actually ask this. Eg, in addition to @Glen_b's point above, if $Z\\sim\\mathcal N(0,1)$, then $Z^2$ *must* be $\\chi^2_1$.
Feel free to retag this more sensibly!
Do you have a large matrix where rows are users and sports are columns and values are measures how much someone likes a sport?
Thank you, regarding (2) "*Does that mean it is better to leave one of them out or merge them?*". My question on the possibility of 'merging' refers to re-packaging the members of `rank2` and `rank3` to a new category `rank23` that is a union of the former two. So, I did not understand your response on this.
Do the functions from my last edit work? Can you try them?
It depends. If you are forecasting the number of years until X occurs, then the zero is meaningful. But if you are forecasting the year on the Gregorian calendar in which X occurs, then no, it makes no sense to treat zero as meaningful as you could have used the Islamic calendar just as easily and got a different result.
What happens if you leave out just one of them?
(+1): Did not know about conjoint analysis before (just read this article:http://en.wikipedia.org/wiki/Conjoint_analysis). Question: So basically one collects preference/purchase data for all or a well chosen subset of combinations and then performs an analyis afterwards to determine the variable importance. Is this correct ?
If $\\rho(X,W)+\\rho(Y,W)\\gt 1$ this relationship obviously cannot be true in general. Such a relationship has no natural connection with properties of correlations--the nonlinearity of the function $(X,Y)\\to XY$ indicates there should be no general relationship between correlations of $X$, $Y$, and $XY$ with other variables--so it most likely is accidental. It would be interesting only if it persisted across several sets of random variables with varying correlations among them.
Cube root. But that means `sign(x) * abs(x)^(1/3)` or the equivalent in your favourite language.
You provided no info about your models. Please show syntax of both commands.
Nice! Upvote ;)
I think this question is about _confounding_, not collinearity. It would be good to know more about the distribution of the covariate (e.g. mean, sd) in each group. The correlation values you reported are not enough information, mainly because the numerical values assigned to the levels of the binary variable in the calculation are arbitrary, making the result not particularly meaningful (and is a reason I wouldn't call this collinearity, per se). Related to your other question, a differential effect of the covariate on the response in each group points more to _interaction_, not confounding.
Thanks guys, I tried searching for this but was having trouble coming up with the right queries.
In your example, you don't check for an interaction.  If there is a strong theoretical reason that establishes there is no interaction, or if the data are sufficiently strong to establish that there is no interaction, then the effects of the two factors are independent of each other, and you can just as well run 2 one-way ANOVAs, with Tukey tests etc.
(-1) This answer uses "consistent" where "unbiased" is needed: see http://stats.stackexchange.com/questions/31036. As such it only repeats the statement that is to be proved and sows additional confusion.
The other difference is the factor 1/n instead of 1/(n-k) where n is the length of the series and k the number of lags. This is to ensure that the autocorrelation matrix is positive definite.
(My pleasure chl - thank you :) )
@Pyca It sounds like an inappropriate use of comments there. You should post a new question, with reference to this one.
I added an update for you above. I've had pretty nice curves (I guess that's subjective) with `method = "svmRadial"`. I hope to evaluate some others, including `method = neuralnet`, though in past attempts I've found the neural network *very* sensitive to `set.seed(). In short, we're trying to 1) better understand the relationship between inputs/multiple responses and 2) make best guesses at the most likely design space to yield desired properties so that we can run new, more focused DOEs vs. a more typical expansive/shotgun approach in terms of input min/max levels to try.
Beg pardons if my writing is not clear, it's a midnight in Lithuania.
Possible duplicate (or special case) of http://stats.stackexchange.com/questions/29271/interpreting-residual-diagnostic-plots-for-glm-models or http://stats.stackexchange.com/questions/44643/glms-diagnostics-and-which-family, although neither of them have answers that will really solve it for you.
3 tables helps.  What are the 2x2 (rater can give diagnosis/can't X what?)?
Is `seq` in your code a typo for `sex`? If you've copy-pasted your analysis code, that might be part of the problem..
Could you explain what the analog of matrix multiplication might be for a "three-dimensional array" (presumably, a rank-three array requiring three independent indexes)?
Actually it's not unreasonable that movies with numbers in the title do betters than movies without, because numbers are generally indicative of sequels to successful movies with pre-existing fan bases compared to movies that have to find a new audience from scratch with an unproven concept.
@nsw, I would like the intercept parameter, or the mean, to be estimated in my ARMA models.
Sorry, I don't have a final answer to your question but I would like to comment.
Have you thought about quantile regression as an alternative approach? It is distribution free, and you could handle correlation among units by using linear quantile mixed models or, alternatively, marginal quantile models
[cross-posted](http://metaoptimize.com/qa/questions/10659/why-we-need-sparsity-for-auto-encoders) to metaoptimize.com/qa/
Seems I misunderstood your question - so the data is aggregated over the hours of a day, or or the days of the week for a number of days or weeks? In that case it is a kind of histogram.
$P(y)$ is not being ignored but treated as a constant because it is a function of the _data_ $y$ which are fixed for the problem at hand. If $A(x) = cB(x)$ where $c$ is a constant (meaning not dependent on $x$), then we can write $A(x) \\propto B(x)$ which simply means that $\\frac{A(x)}{B(x)}$ is a (unspecified) constant. Note that the extrema of $A(x)$ and $B(x)$ occur at the same locations so that things like maximum a posteriori probability (MAP or MAPP) estimates can be found from $P(y\\mid\\theta)P(\\theta)$ without the need to know (or compute) $P(y)$.
it's also going to get hammered on SO because it's just a "how can I?" question, rather than a specific programming question ... Can you give a little more context? I would be tempted to do a quasi-Poisson fit (i.e., fit the model as a Poisson lasso, e.g. with the `glmnet` package, then make a *post hoc* adjustment to the standard errors of the parameters based on the estimated residual deviance ...)
Can I consider the correlation between the numbers of users and the average of lines of code in a project as a meaningful result? thanks.
Have you examined the OLS output after re-sorting your data?  (If not, you might consider trying it: just permute the cases randomly or sort them on some other variables.)  Only the things that change in the output will be of concern to you (insofar as the sorting goes) :-).
Do you have individual level data and can determine which individuals were eligible for mammography in either period? Or do you just have aggregate #s of people and mammograms?
One suggestion about seeing which model is suited to your situation is to simulate from the fitted model (and exponentiate back for the log-transformed one) and see which looks more like the original data. (There are other, faster ways, but they're less easy to explain in a line or two.)
Most likely by `the change-of-variables technique` is meant the usual method involving Jacobians. However, the result that you are asked to prove is false unless the univariate standard normal random variables are **independent** random variables.  See [this answer](http://stats.stackexchange.com/a/30205/6633) for a great description of how two normal random variables can fail to have a bivariate normal distribution.
Not quite sure why my formulas aren't rendering on the question page as they render just fine in the preview. If someone has enough reputation and wants to edit so that they display correctly it would be much appreciated.
*Many* probability distributions have power series expressions like this one. That does not mean they--or Bessel functions--take much time at all to compute accurately. However, you have much more fundamental problems than that. First, $n$ must be an even integer so that its values will be nonnegative.  Worse, the integral of $I_n$ diverges and so cannot be rescaled to form a proper PDF. Could you perhaps edit your question to explain why you think you have data described by a Bessel function?
By reviewing a suggested edit for this post, I noticed the following request: "a citation/reference justifying these would be great if anyone could provide them".
Thanks for your help, appreciated!
Oh, although whuber raises a valid point. This would not help you decide if a value is part of your set, it would only allow you to calculate the probability for the value given that it came from the same set.
@Jeromy, thanks -- the thread was helpful, but my first question is whether it is defensible to average within subjects and then compute a between-subjects sample SD. This makes the effect size double what it would be if I simply used a pooled sample SD estimate treating each data point as if it came from a different subject.
Google search: "spurious regression in econometrics". Or check out [Granger and Newbold's paper](http://wolfweb.unr.edu/~zal/STAT758/Granger_Newbold_1974.pdf). Others may supply more details in answers.
How do you sample from each vector? If you only sample the means in each of the GPs you do not take into account the variances. Otherwise you will have to devise a sampling technique that is consistent.
@Macro: If the distribution of the predictors is exactly the same for each level of Y, does this mean that the proportional odds assumption does not hold?
PCA not PAC ;) (*15 chars*)
If you are happy summing item scores, you have already assumed more than strictly ordinal level of measurement. Strictly speaking, ordinal measures cannot meaningfully be added or averaged (incidentally, Stevens is clear about that). Once you have done that, treating the resulting scores as interval level data is perfectly reasonable.
Do you just want to put all numbers that are 1 unit apart into the same cluster? Are you just looking for code / how that can be done in R?
hint: [$\\mu_3$](http://en.wikipedia.org/wiki/Skewness#Definition) is the [third moment about the mean](http://en.wikipedia.org/wiki/Standardized_moment), not just the third moment. $E[X^k]\\neq E[(X-(EX))^k]$ unless $E(X)=0$.
Imagine an example: your dependent var is dichotomous, two groups. And the scatterplot of these data in the space of some two variables X and Y shows that the groups are two separated oblong parallel clouds (like "="). You do PCA of X and Y variables and obtain PC1 (stronger one) parallel to the two clouds and PC2 (weaker one) perprendicular to it. Well, now, you see that PC1 is unable to predict (discriminate) the groups, but the weaker PC2 can do it well.
You can use the same table as for finding p-values, but you look for the probability value in the *body* of the table, and then read off the Z that gives that value. If you can convey adequately what is tabulated in your tables and how it's organized (there are images of [tables on line](https://www.google.com/search?q=normal+distribution+tables&tbm=isch), if you find one that's effectively identical to yours, a link would suffice), then I'll try to come back with more details.
+1 I think you have it basically right. You might be interested to know that the target of an estimator does not necessarily have to be a particular "parameter" of a model: it can be *any* property of the model, such as a function of its parameters. For instance, $\\mu^2$ is not a parameter for a Normal$(\\mu,\\sigma^2)$ model, but it can be estimated.
@CodeGuy I changed the question and answer accordingly.
A better fix would simply be to make `subject` a factor in `data` before melting.  Then `subject` will automatically be a factor in `data.long`.