tttjjj/elt_bench huggingface dataset:
To get a sense of how this recommendation might play out, consider a case where there are 100 (say) important predictors (highly correlated with the dependent variable) and the rest are completely unrelated to the dependent variable and to each other.  Retaining those with p-values less than 0.2 assures that you will end up with approximately 100 + 0.2*(2000-100) = 480 variables, of which 380 are worthless.  For *small* datasets this approach is sometimes used as a quick initial screen, but it really cannot seriously be considered here.
...Im thinking of **adonis** {vegan} analysis which is a non-parametric version of **MANOVA**
Indeed, and this is why the error terms exist. In general $\\hat A_t$ never equals $A_t$, this is a core fact of any estimation procedure. If for some reasons you _need_ to obtain this equality, then one interesting avenue is to devise a way to _allocate_ the error of each period over the estimated coefficients (that's one way to obtain "time-varying" coefficients).
Also, for an exceedingly gentle "ramp" see the current video lectures on machine learning by Andrew Ng at Stanford. They provide a fairly sound introduction to Hastie or Bishop. http://www.ml-class.org/course/class/index
I can't add because the output is too long to be fitted inside the box provided by stactexchange.The results from the output are very long. Whenever i add the all the data in the stackexchange warns me about the number of characters left.
So the number of particles stays fixed all through filter predictions for $y_1$, $y_2$, .. etc?
It sounds ambitious bordering on contradictory, the aim to do clustering and then use the outliers in clusters, in a scenario where it is unclear what would constitute a natural cluster. For your aim (attacker mimicking e-mail) I think you are better off thinking about different approaches.
I edited the maths in your question, and changed it from $(X^T)X$ to $X^TX$ which is hopefully what you wanted. Or was it $(X^TX)^{-1}$?
The idea of using a logistic transformation is a good one, but this kind of regression model has problems (as explained in my answer).
Ah yes, sorry I misread your answer. Thanks!
Of course it only occurs when it's not of full rank. Look at the definition of PSD (and @sjm.majewski's mention of the relation to variance), and this much is clear. But to define it as pathological seems odd, since these non-full rank situations happen all the time in practice. This is not simple pedantry - it affects real data sets every day, and as a result generates regular questions here. I'll talk about missingness and pairwise deletion above, because there's not room for it here.
Firstly thanks, you answer clarify me on method of standardize both test. However pls explain the first method "work cut out for you", is it means I use raw score with 'assumption'. How if i use likert  scale scoring method (Thurstone),how to compare both test then?
There's a PDF copy available for download at the link above if that helps.
@svadali I did not replace 365 by $n$; I replaced it by 2, which gives a simple problem we can check by hand.  At any rate, Step 2 has a typo and the reasoning in Step 3 is incorrect.  In fact, the approach is flawed because it does not consider various complications that arise with multiple "collisions," such as two people sharing one birthday and another two sharing another birthday.  I get the value of 6/16 (four people, two possible birthdays, at most two per birthday) because of the 16 possible configurations, 6 have two people share one birthday and the others sharing the other birthday.
If it doesn't work out, please let us know, because there are other possible approaches available (such as rewriting the system to make $(x_0, x_1, \\ldots, x_n)$ a fixed point of a multilinear operator).
Regarding your second point, if you have an estimate of scale reliability (test-retest measurement, or internal consistency--i.e., Cronbach's alpha), it can be used to compute the [standard error of measurement](http://bit.ly/OQo7ZB) of your scale scores.
1) that's why I wrote "orthogonal". 2) yes, gram-schmidt. 3) I added the pseudo code. thanks
@DimitriyV.Masterov It's less used so it's harder to generalize. From what I've seen, it's pretty common to use a log-link with inverse Gaussian, but other links may be suitable in some situations, such as an inverse link.
Post everything. Post your CFA output and the full SEM output. Post your path diagram. Without these, it is impossible to even begin to figure out what's going on with your data. Generally, it makes no sense to start with CFA, and then convert some indicators (i.e., dependent variables) into explanatory or mediating variables.
Can you put the name of the article please? It seems to have disappeared!
It might help to consider $p = 1$ first. In that case, $\\hat{y} = \\hat{\\beta}_0 + x \\hat{\\beta}$, with $\\beta_0$ the intercept. This is the equation of a line passing through $(0, \\hat{\\beta}_0)$. Extensions to higher dimensions are immediate.
I am adding the `[homework]` tag not because it is *your* homework, but because the question meets our general guidelines for when this tag should be added.
@MarcClaesen I can't see how it relates to what I'm trying to do here.
try `?splinefun`
Heh, thanks. You write very well.
... but assuming my rather equal SE between groups, and my sample size of 113 (56/57), it is possible to do not rely on CI95% comparisons and use the Z-test approach? Thank you again, Nick.
Good points. I was using "what the instructor wants" in both the narrow sense (will your project be evaluated on this basis) and in the broad sense (what is the goal of the project and the nature of the question). In my experience there are two reasons that people write reports -- because they are told to or because they want to communicate something. Based on how the question was written, I put this in the "because I was told to" camp.
any finite sample is 'small' - the law of averages refers to expecting that asymptotic (i.e. infinite sample) results about random numbers should apply in finite samples.
Haha, yeah I guess the authors of the package didn't take the time to do spell checking.
Thanks! For standardized moments, is its MLE  also obtained by the invariance of MLE, and how about its unbiased estimates?
I added the quotation from the book I am using
Let's reverse your argument: if you don't like the advice here (and the implicit stance that terminology from mainstream statistics can be used or borrowed), no offence taken, and I see two other possibilities for you. 1. You invent your own terminology because no terminology you know of appears appropriate to you (no sarcasm or irony; that's how all terminology starts). 2. You spell out what you are doing in a new question with much more detail and seek advice specifically from machine learning people or more generally anyone versed in neural nets.
I haven't read the papers you refer to, but I would guess they mean that the mixture in question constitutes the different classes. Ie, if you had 2 classes, you would have a mixture of 2 (possibly multivariate) Gaussians. So although a sufficient number of Gaussians can approximate any distribution, that isn't what they mean.
@Glen_b I clarified my question. It should be clear now. Thanks for help.
Try `sample`ing before you plot. I doubt you really need all 200,000 points at once.
Could you tell us where (anywhere in the world) we can find Census data with *names*?
That was actually what I meant with the GLMM. model <- glmer(capture.rate ~ species + (1|container), family="poisson")). That would at least account for the dependent samples. I forgot about the fact that I can use different distributions so the whole normality assumption won't matter. Thanks
The natural link for Poisson regression is the log (and offset is addative).  You want $E(x) = log(\\lambda t) = log(\\lambda) + log(t)$  The intercept will then be the log of lambda since you set the offset (with coefficient 1 as you say) set as log(t).
Yes! Many posts on this site. One keyword to search for is L1.
Here is an interesting paper on how to better structure NN's together to solve 2d grid problems like image classification.
http://axon.cs.byu.edu/~martinez/classes/678/Papers/science.pdf
Math formatting is accomplished with mathjax, which is basically the same as LaTeX. Numerous tutorials and code-to-symbol lookup tables are available online.
In my case, I actually don’t care about the actual values for probability or likelihood. I am evaluating two models that are identical except that they use two different probability density functions.

For example, if the variable at issue returns a value of 1.0, and f(1.0) from one PDF is 0.3, and from the second PDF is 0.2, I am wondering if the first model “wins” to a degree of 1.5. The 1.5 would be considered relative and would be compared to calculations at other observations.

It is not all that complicated, and there are no scaling issues since the models use identical data.
Hey Glen_b, Ok can I get an explanation or example of how to carry out your suggestion and get a reasonable plot? Thanks so much for your help. Really appreciate the time you put into this.
What has continuity to do with it? Why do you think that's a good reason to make it a factor?
A person with experience in using Normal approximations to the Binomial would proceed a little differently: they would apply the (usual) [continuity correction](http://www.cs.uni.edu/~campbell/stat/prob9.html), as in `1 - pnorm((30-0.5)/50, mean=0.5, sd=sqrt(0.5*(1-0.5)/50))` (this is an R expression), whose value is 0.1015, in quite close agreement with the Binomial cdf.
As far as I know, you _can_ test a clustering hypothesis, see http://www.stat.ufl.edu/~casella/Papers/paper-v3.pdf or the SAS manual: http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_introclus_sect010.htm. My problem is here that I have both variables and time points.
What is MAP criterion? Please de-abbreviate it and tell more about it, OK?
@lnes, `glmer()` always uses Maximum Likelihood (ML) rather than REstricted Maximum Likelihood (REML), see [here](http://glmm.wikidot.com/faq#reml-glmm).
ANOVA *is* a linear model. For repeated measures, people may have been suggesting a multi-level model.
No No it is not a homework .. believe me. It just a question that came to my mind.
Are you sampling randomly and separately from $D$ and $D_2$? I don't see how you'd get a simple sample of $D_1$ by subtracting points in $D_2$ from different points in $D$...E.g., if all points in $D_1$ happen to equal 1, you won't get $D_1$ by randomly sampling a 6 from $D$ and subtracting a randomly sampled 3 from $D_2$...you'd have to sample the corresponding 5 from $D_2$ to be sure of getting a data point sampled from $D_1$. I.e., I think you'd only get to sample randomly from $D$ or $D_2$, not both.
No, those coefficients are for the *orthogonal* polynomial terms: you've written the model for *raw* polynomial terms. Replace $X$, $X^2$, & $X^3$, with values of $L$, $Q$, & $C$ respectively (from the look-up table).
$\\hat{y}_{t+h|t}$ means "the forecast at time $t+h$, given data up to time $t$".

So no, it's not the next-period forecast (unless $h=1$), it's the forecast for $h$ periods ahead from time $t$, using information available at time $t$.
@Srikant: Thanks for the link to the notes. I am actually trying to detect (D) [I have separate tests for (B) and (C) which I run before (D)] and from that perspective it does seem like one test is reverse of other. Other than that, do you also have a comment about the procedure itself that I was thinking of using?
Probably should label as homework.
TRhanks a lot Maaten :) I would upvote your kind reply as soon as I gain enough points to qualify to vote (not that you care, of course!) :)
Why did you normalize the features/predictors?
As far as I can tell, then, asserting there is an "unknown bias" has no information content, because infinitesimally small amounts of bias are indistinguishable from lack of bias.
Thank you all for your replies. It is clear to me now! Much appreciated.
@Wern An adaptive/autoregressive component exists when the "predictors" includel either memory (previous values) of the endogenous (Y) series or exogenous series OR lagged values of the error term. It is important to identify and model , with possible  lead,contemporaneous and lag effects around EACH of the Holidays. Furthermore there may be assignable cause to specific days-of-the-month , dependent on the day-of-the-week. For example the 5th of the month may be important GIVEN that it was not a weekend. We have been able to successfully model/automatically identify this structure.
Great, Thank you so much!!!  Just requested the books from the library :)
Thanks ! In the original data (~18000 obs) the minimum cluster size was 10 and there were 4 predictors for CTP in the predictor matrix. I am now imputing all variables and I have obtained a smaller dataset (2800 obs, 15 vars) which is still causing R to crash - I have updated the question with a reproducible example.
Edited the question with another, well is kind of homework self-study.
Answers can be found under the [tag:errors-in-variables] tag and the [tag:pca] tag.
For commercial reasons I am not allowed to share the code, and for ethical reasons I am not allowed to share the data (this may change). Saving to a mat file means only Matlab users can interpret...
zero-inflated: modelling data that has more zeroes than you'd find in a Poisson; zero-truncated - modelling Poisson-like data with zeroes removed (whether you took them off or because the zero cases are just not observed for whatever reason).
We'll see if others feel differently, but to me those correlations are not high enough to be concerned with multicollinearity, nor are the VIFs.
@Nick has already pointed out that you have not stated Bayes theorem correctly in your model.  More pointedly, since $P(D+)$ occurs in both the numerator and denominator of the expression on the right side of your displayed equation, your model simplifies to $$P(T+\\mid D+) = P(D+\\mid T+)$$ which is surely not what you mean!  So what _is_ your model?
I don't know of any name for the concept nor any general way to find examples.
Could you indicate *where* in the Jager and Wellner paper a full definition is given? As far as I can tell with a quick skim, they do not provide a definition but merely refer back to the Csiszar paper.
Thanks a lot! I will look into Van Breukelen (2006).
Does nobody have an opinion here?
Anyway, try to remove a layer and/or avoid training algorithms that use second derivatives, because usually they take a lot of memory (e.g. MATLAB trainlm).
@suncoolsu: I've updated my answer. Does that help?
@aaronsw you've got too many questions for 1 question I think.  I see your 3 questions as: what is a credibility region for a point estimate of the proportion of people with the treatment who die (which is what you have done the frequentist confidence interval for, and I've given that); what is a credibility region for the difference between the treatment and control proportions (which reduces to the  distribution of one Beta distributed random variable minus another Beta distributed random variable - if you want the exact distribution it will be complex); and how to do the hypothesis test.
what do you mean by "gives you a double"?  Are you referring to estimates of the two types of classification errors that can occur in a two-class problem?
Do you mean that you wish to update the mean and variance - that is, given a computed mean and variance on the first $t-1$ observations you want to compute them on $t$ observations? Are these univariate variances, or variance-covariance matrices? In the univariate case, you only need store a few values.
I can see why you'd be concerned; that website is inscrutable! Don't they mean "e.g." instead of "i.e."? And those response rates (2.5, 1.3) are percentages, right? 2.5% and 1.3%, respectively?
(+1) This is basically a good summary--although it would be nice to see some abbreviations explained--but there is one small technical fault: for the chi-squared distribution to be valid, the exponential parameter has to be fit using maximum likelihood rather than this approximate method of moments. It turns out also that your result is fairly sensitive to the *post hoc* decision to combine the last two categories. These increase the $\\chi^2$ statistic to $5.402$ (based on an estimated exponential rate of $\\hat\\lambda=0.02556$), corresponding to $p=0.37$--still not significant by any standard.
@Kaish Are you *sure* the table is giving you $\\alpha$'s rather than upper tail areas?
Please search the site before posting a question here. This is a frequently asked question that has been addressed many times.
I clicked on the link to see "Asymptotic theory for parametric models" and it was broken.
+1 nice references. So perhaps the answer is: "there's no point constructing strong learners when you can construct weak ones more cheaply"?
Your friend's description is incoherent: it sounds like there are only $11$ "jars", one for each Browns win. If you put the Bengals wins into those $11$ jars, the Pigeonhole Principle guarantees there will be jars having wins from both teams. But so what? The burden is on your friend to show how this "jars" metaphor--however s/he manages to fix it up--is an accurate model of how games are actually scheduled and played. For instance, they should be able to tell you what the jars are meant to represent. Are they all Sundays? Sundays when at least one team plays? Both teams? Are they games?
(+1) Indeed, the unbiased estimator you describe is the minimum variance unbiased estimator. :)
@chl, if your swap partition is larger than 5GB sure :) If you know that you will need more swap than your current swap partition size, you can mount additional space as swap. This needs manual intervention of some kind, but on other hand it is doable via R.
Yes, just use the ANALYZE > DATA IMPUTATION function...
I'll post some R code giving an example.  The negative hessian is PSD at the mode(s) but there may be more than one mode.  Depending on where you start the algorithm you get different stationary points.
See also [this question](http://stats.stackexchange.com/questions/17294/is-my-data-distribution-normal-tried-shapiro-and-kolmogornov-smirnov-tests), and [this one](http://stats.stackexchange.com/questions/73030/how-can-the-kolmogorov-smirnov-test-be-used-interpreted) ... there are many other questions and answers here that are relevant to aspects of this issue.
giggle. Funny mnemonic. Which may make it more memorable
I should emphasize that I'm talking about estimating P(Y|X) via stochastic gradient descent, in which case the order of the training samples can influence how fast or whether it converges to the correct model. I'm not just using sample averages, where the order of the samples doesn't matter.
The same principles apply, Flav, but to fully write out the calculations requires some form of higher-rank tensor notation.  Do you really want that?
$N_1$ as described in the question is not a random variable.  It's a fixed but unspecified value.  If you mean "what is the distribution of the number of ones obtained in 100 trials," then yes, that is binomial with $n = 100$ and probability of 'success' equal to $p = p_1$.
@B_Miner: You're welcome. If you have lots of training data, and a seriously time-varying system, then you might want to train with only $k$ instances after validation. In my case, I usually want to use every last scrap of training data, but need to use less to get out-of-sample estimates.
It depends what you want to do. Please give more information about the purpose of your analysis.
That wasn't me.... You say _phase offset_ as if that named what is going on, and it does mathematically. But for you the key point is more likely to be that Dec 31/Jan 1 is an arbitrary origin for time of year given lags in temperature response to variations in radiation receipt. So phase offset is a name here for something climatological too, the timing of minimum and maximum temperature relative to your recording system. (It's a minor detail but I prefer quantifying time of year for 12 months as 1/24, 3/24, ..., 23/24.)
+1 This answer is correct. If you believe the variables are not independent, then your *question* is wrong, not this answer, because the question implies the variables *are* independent.
I tried to get materials for MCA by googling. But didnt find anything much on it. By any chance do u know any link where i could find much materials on the same so that i can able to convert nominal data into coordinates as you mentioned for k-means and such algorithms.
@Greg... Thanks! I will take a look at the ks.test and vis.test.
In your bottom line it depends on what you mean by 'easier'.
Yes you can, for example the linear probability model for any 2 state categorical variable taking on 1 or 0
Why do you want to know the answer to this question? Suppose the correlation is stronger for men than for women. Then what?  Unless it's in a regression framework, I don't see how the answer can help you do anything
Did you receive a response?
@rpierce, if the difference between observers is constant (does not interact with genotype) then that difference will be included in the differences between the 2 controls/wildtypes and will be adjusted for in my description.  But if there is an interaction between observer and genotype (one observer sees a bigger difference) then that will be confounded with the genotype interaction of interest, hence the need to understand and accept the assumptions.
`Do I have to change the name from nMDS to MDS if I use euclidean distance, as euclidean would be a metric method` If you are speaking about input distances when you say "Euclidean" or "nonEuclidean", the answer is no. Difference between metric and nonmetric MDS is not about the input distances, it is about the allowed transformation of them ([see](http://stats.stackexchange.com/a/31291/3277)). Keeping the transform strictly linear results in metric MDS; allow nonlinear monotonic - nonmetric MDS.
"*could be up to 300 for highly skewed distributions*" ... in some cases, it could be a heck of a lot more; or it might never happen. Pick any $n$, and I'll show you a case where it's not enough.
I have a similar question at http://stackoverflow.com/questions/21686200/min-distance-algorithm-in-r , I downsample in order to plot the "fewest necessary" points from a huge data set.
I see, thanks!!
I'd look into 'robust' methods. These methods try to de-weight outliers. E.g. there is a [robust spline](http://rss.acs.unt.edu/Rdoc/library/aroma.light/html/robustSmoothSpline.html) algorithm in R.
There's also RPy http://rpy.sourceforge.net/, R as a library for Python, so you get the up-to-date, proven-error-free aspects of R with the syntactic simplicity of Python.
For a proper answer a bit more information on the data (time series, cross section, panel?) will be useful. Generally, in the case of simultaneity you will have to use an instrument which exogenously changes "IPO underwriters" and which does not affect "IPO first-day market returns". Otherwise you cannot separate the simultaneous relationship and identify the correct coefficient for your independent variable.
The convergence of the PDFs does not seem to say much. For instance, you could mix in $1/n$ of a distribution with PDF proportional to $(1 + (x/n)^2)^{-1}$ with each $t_n$ distribution and still arrive at the same limiting PDF, but all the while the distributions in the sequence would be getting *fatter* tails. The possibility of subtle behavior like this makes arguments based on limits of PDFs less than satisfying. Besides, doesn't the question really ask about *small* degrees of freedom?  It wants to know why the sequence "starts out with the fat-tailed shape that it does."
@Glen_b are you putting the dependent variable on the right hand side? Typo or maybe I'm confused:)?
+1 I liked the other approaches section!
@Adam is R^2 calculated from the testing dataset or is it using the pruning set to calculate MSE?
Tank you for your answer, ThomasH. I leave the question open just to see if anyone can address the rest of the question.