tttjjj/elt_bench huggingface dataset:
You may have too many predictors for your 200 data points.  Your multiple R may be large, but the large value may be an artifact of the relatively large number of predictors you are using.  (Look up "shrinkage" or "capitalizing on chance" in any reference on multiple regression for more details.)  How many predictors do you have in all?
FYI As someone suggested, I added a small value (between 0.0001-0.0009) to every datavalue. It helped to get rid of the singularity. Thanks for your help
+1 for the article, it is a truly great article, IMHO :)
It's possible the OP is being overly cautious, but I would suspect these books are more advanced for someone who is asking about "for dummies" books.
@cardinal - I thought it was a ratio of independent normals *with zero mean* was cauchy with zero median and scale parameter equal to ratio of the normal standard deviations.  If they have non zero mean then it is not cauchy.
I can't download that article, care to upload it somewhere else? Anyway, how would such a model be constructed? The OP is curiously silent on what she actually wants to do with the variables
Sounds like you're asking about *[imputation](http://stats.stackexchange.com/tags/imputation)* of missing data. You might like to start [here](http://en.wikipedia.org/wiki/Imputation_%28statistics%29), and maybe take a look at [this](http://www.stat.columbia.edu/~gelman/arm/missing.pdf). Many stats packages have a variety of methods available for such things. Those also give you some handy additional search terms.
Absolutely right @Michael Lew. My example was not very didactic because this is an extreme case. Welch's t-test comes in hand in borderline cases, like when of the sample has a variance that is 4 times smaller. I simply wanted to highlight that the approach is consistent in the limit.
Gavin, thanks for your feedback. I did not pre-scale the design matrix, but I did apply scaling via the 'scale=TRUE' argument. I assumed (wrongly, it seems) the beta values would be automatically re-scaled for me. Is best practice in this case to re-scale manually, i.e. divide the scaled beta vector by a corresponding SD vector?
I'm not familiar with them so they didn't cross my mind, but after having a quick look at the wikipedia page they may be just what I need. Thanks for the pointer!
What statistic is it you want to bootstrap exactly - are you wanting to bootstrap the entire 10x10 correlation matrix between the two 10-column sets?
+1 good catch.  From the 2D histogram, my first thought was that it was heteroskedastic.
[Two sample Kolmogorov-Smirnov](http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov.E2.80.93Smirnov_test). If you calculate $D$, you'd have *a* measure of dissimilarity.
Alright, chalk another point up for me getting my act together and sweaving everything.
@onestop: hmm, think definition of O/E == LR with the log forgotten above? I agree with what you say about Cox PH -- that's not the question I was trying to answer, but your advice is better in the broader context.
@Feras, you may have a perfectly valid reason to do this. If you do, can you share it with me?
Thanks. I'll pick up a survey sampling book.
Welcome to the site, @user28298. It is not clear to me whether you are looking for help understanding the statistical ideas here, or w/ the R codes. If it's the latter, this Q would be off-topic for CV (see our [help page](http://stats.stackexchange.com/help)), but on-topic on [Stack Overflow](http://stackoverflow.com/). If you have a substantive statistical question, please edit to clarify; if not, flag your Q & we'll migrate it for you (*please don't cross-post, though*).
I want a robust measure for the different columns, i.e. to aggregate the values in the different columns to make a robust summary
This is called over fitting
If you get the same values all the time for LAbase, there is no way to do anything.
Indeed, thank you. It would seem  that the approximation does quite well in the right tail of $F$; it might not be unreasonable to expect it to hold for $Y_1$ as well as many cdf's are also linear in the left tail.
[second equation on pg 23](http://cis.temple.edu/~latecki/Courses/RobotFall07/PapersFall07/andrieu03introduction.pdf). If you condition on the children, you have to take them into account as well, so you can't just write what you have there.
I think this is the sort of model I am looking for. It looks like I'll need a parameter for the ability of each person. Perhaps I should explain the actual problem I am solving. I am looking at individual soccer players. Each test is a batch of matches they played in, with similar conditions (all home, in one season, in a league match, say), and the response variable is whether they are "awarded a card" (yellow cards and red cards are given for fouls). I think some variant of the item response model will probably be useful.
Well, for starters, frequently relationships are not linear. Unless I had *a priori* reason to, I'd usually only do it if the residual analysis suggested the linear model was clearly inadequate. On the other hand if I had no *a priori* sense of what form the relationship might take I'd fit something more general from the outset.
Graphical models may be similar to what you want: http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html
There is not enough information disclosed in order to provide an answer.  You need to explain your data and what your research question is. Further, sample-size estimates should be made a priori and not a posteriori. Have you researched publications in your field to review how others have set sample size?
What does saturates mean?  Does it mean that the slope of increase is going to zero?
You are using the correct package. The "other standard" LME models package in R, lme4, does not allow for highly specified covariance structures. For the matrix structure you describe what you want to take advantage of is the type $corSymm$ and not $compSymm$. So you will code up something like "*corSymm( value=c($\\rho_1$, $\\rho_2$, $\\rho_3$, $\\rho_1$, $\\rho_2$, $\\rho_1$), form= ~1| GroupingVariableX)*".
"treat Likert-type rating scale as interval" is just think that it is interval and use it as interval. Yes, it implies that you set measurement level to Scale.
@AndrewCassidy Is it possible to add a parameter to take `AIC` in account on the build-in function `stepwisefit` of matlab?
@Procrastinator is right, you should not multiply the posteriors. Instead, put all 5 datasets and a single prior into one analysis (straightforward with BUGS/JAGS etc) and obtain a single posterior from there.
Can you give us a better idea of what you have to work with? For example, are you just being asked to *design* a project, or are you working from a data set, etc.?
+1 I think this is a great idea given the OP's constraints.  All one can hope for is an approximation, so the trick is to pick bins that have a high likelihood of being narrow and containing the desired quantile.
Thanks @AnthonyDamico, but my problem isn't only in `survey()`, but is also in identify the sample design correctly. I know that in all cases the pontual estimation will be the same (at least, for the many I tried), but I'm also interested in the standard error estimation.
@Mahoni What you say doesn't sound quite right. You have a convex optimization problem here, so it shouldn't matter where you start. I would suggest checking your code and your derivations again. 

Also, the optimal $\\beta$ that you state is optimal only for the unregularized convex optimization problem and not for the regularized as in your case.
A few questions: (1) "level of disclosure" and "index" imply a variable with more than two levels; have you considered using an ordinal scale as the dependent variable? (2) Does disclosure vary over time or is it operationalised as constant over time?
I just wanted to know how its done so I could apply it in my current situation!
See discussion about avoiding [dynamite plots](http://stats.stackexchange.com/q/1173/1036). The idea could be amenable though to the current question - just plot the error line to show whatever range of the reference distribution one is interested in, and then plot a point to show the current users value in relation to the reference distribution.
@GentlemanEddie "looks better in terms of significance". There's no need to play the significance game: http://mchankins.wordpress.com/2013/04/21/still-not-significant-2/
I did use the gap statistic if that is what you mean
@user40124 alto's answer sounds right if one takes the process of increments as the "observale" process
Could you please ask a specific question, @bety?  Otherwise this thread will have to be closed.
The plot on the page you point to has 10 vertical lines.  You want to do that with 15k lines?  A typical display is <2000 pixels wide.
"The researcher armed with a confidence interval, but deprived of the false respectability of statistical significance, must work harder to convince himself and others of the importance of his findings. This can only be good."  Michael Oakes
+1 for pointing out a great article I didn't know about. (Also for some much needed skepticism about the utility of Mathworld's view of statistics).
I think you're being a bit picky about the non-constant variance.  It appears ok to me.  What is the purpose of the regression? Explanation/hypothesis testing or prediction?
Is this for a course or a homework assignment? If so, please add the self-study tag!
Hi James, thanks a lot for your reply. Just to clarify, "M^TV" stands for the transpose of matrix M multiplied by V, right? I am not enirely sure I understand the details, though. I don't want to take up too much your time, but if you happen to have a simple example at hand, I'd be grateful. Based on what you have written above, the fact that the data plays out so nicely in the paper appears to be a bit of co-incidence.
No, I am not asking for the optimal number. I have already decided on the number: 5. I want the algorithms to use to be able to divide my data into 5 bins according to their distributions.
Of possible interest: [More efficient plot functions in R when millions of points are present?](http://stats.stackexchange.com/q/7348/930), [Visual Analytics of Large Multi-Dimensional Data Using Variable Binned Scatter Plots](http://bib.dbvis.de/uploadedFiles/20.pdf) or [Variable Binned Scatter Plots](http://bib.dbvis.de/uploadedFiles/300.pdf) (PDFs), or [Hexbins!](http://indiemaps.com/blog/2011/10/hexbins/).
It is often a good idea to take into account pairwise interaction. But not all lof them:  You need to think trough which makes sense!
Why do we always pick on the gingers? :)
Glen_b, I'm using it in order to compare if distributions of values differ from each other or not. More specifically, I'm studying the length of introns between different organisms and want to see if the introns from one organism are significantly longer/shorter compared to another.
A hint: What happens to $W$ in your last formula?
Can you post the data?
Sorry, I tried to be concise. The used machine only can count until a limit (100000) so counts over that limit are registered as "over 100000" and this is the problem, instead of the decreasing curve of the normal distribution I have an over 100000 group which is not very useful.
I have, and I have read 

"Analysis of Variance of Multiply Imputed Data by Joost R. van Ginkel and Pieter M. Kroonenberg"

All i can think of is: run RMA on all 5 datasets and get means of the results. Is this the correct way to go?
I would like to see a more rigorous demonstration of that, @jbowman.  After all, although OLS regression is unbiased, it introduces a negative expected correlation in its residuals, so it's not far-fetched to think that a regression-based ACF estimator might do something similar.  The problem with the present question is that no specific estimator of the lag-1 ACF is proposed, making it unanswerable without further assumptions.
GMM techniques are not unique to econometrics - although other flavors of statistician tend to have other names for the same ideas. They are popular anywhere you want to do statistical inference but can't justify a full modeling approach (or don't want to) - see applications in biostatistics, survey research, social science, and probably much else.
BTW, I'm not sure Fourier will work well on EEG signals. wavelet might do better. http://www.google.com/url?sa=t&rct=j&q=eeg%20signals%20non%20stationary&source=web&cd=1&ved=0CCIQFjAA&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F9639%2F30462%2F01403164.pdf%3Farnumber%3D1403164&ei=iHqoTspR04riBOTJ0Ao&usg=AFQjCNFbRfdiyypaAwiG7E0ESZWnDSROJw&sig2=38gt0ytUb8YGeaxZeWxgRg&cad=rja
Can you give us some rough idea of your sample size?
@Alecos I meant that the correlation between independent variables or should I also remove one of them as well?
@DKK this will not guarantee that the coefficient is positive, which is what NK1 is looking for.
+1. I use this technique all the time.  Occasionally, however, having the formula stored in a variable causes issues.  See http://stackoverflow.com/a/7668846/210673 for use of `do.call` evaluate the formula before calling `lm`.
What exactly do you mean by "batch effect in correlations", and what would constitute having 'taken care of them'?
Great, thank you.  For the second part, I was wondering how the expected value of N_ess/N would depend on f(x), w(x) and N.  Rather than going through the maths, I ended up doing it numerically for the particular forms of f(x) and w(x) that I'm interested in.
R objects *do not* need to be in memory.  Memory mapping is very easy.  300k dimensions is not a problem.  I also presume your data is sparse, as that is the case with nearly all text problems.
I don't have it on my Mac, while I do have other parts of Open Office...
Sounds like you're not fitting a mixed-effects model which you should do here to account for variation between the operators and (nested within that?) the robots. As for the issue with negative damages, you can get around that by doing some kind of transformation
(1) Your basic problem is that you work too hard: almost all this code is repeated at least once. That leads to errors and makes them more difficult to debug.  For instance, did you really intend to have two different formulas for `C.stat`? (I don't think either one of them is correct: you seem to assume the data are sorted by increasing `X`.) (2) What precisely do you mean by "make this function two-sided"?
But assuming he meant what he wrote, what's the point? Even with restrictions on $\\sigma$ he can't estimate both $\\sigma$ and $\\sigma_\\theta$ separately...
5-point Likert-type item are often used, where response categories are "Strongly disag.", "Disag.", "Neither ag. nor disag.", "Ag.", and "Strongly ag.", without explicit numerical score. What's your 6th category? Response coding is immaterial, for regression, data checking, or summary statistics. Any scaling, linking or equating method can take that into account. What's really matter is the underlying measurement model you are ready to assume, because it will determine whether scores are to be treated as sample-dependent discrete scores or a 'reflection' of a latent trait on a discrete scale.
Usually it is expected that you show some willing by saying what you think the answers are or how you would approach them. Otherwise this is just "please do my homework for you".
Excellent suggestion! Thanks Aslan986..
However, I believe this would contribute to the features list. I don't see how we could use this data as a 'datapoint'.
Nevertheless, thanks, I will definitely try this out.
@Vass: The data induces a likelihood.  So, the likelihood is also known.  It has Dirichlet density, which you integrate to give the desired probability.
@Xi'an Yes, that's what Bilal suggested in his 2nd hint.
@user1489975, if you have found this answer helpful, you should consider upvoting it (by clicking on the upward facing normal distribution), & accepting it (by clicking on the check mark to the left of it).
@Ali: If you know your data then you do know your data.  I'm a big fan of "trust but verify" (Walt Flom) and that makes me want to find a way for the data to demonstrate that EM@GMM is the appropriate solution.  If someone put planar or parallelepiped box patches in your space, they would be fit, but a top-hat (or parallelepiped-oid) mixture model might be more appropriate for them.  This is the difference between theory and practice: in theory they are the same, in practice they are different.
You need to input the degrees of freedom as well:
If we assume the range is $0\\le x \\lt \\infty$ and reasonable values for the parameters, then when $k\\gt 0$ is *rational* the mgf can be expressed as a finite algebraic combination of generalized hypergeometric functions.
@MichaelChernick, I recognize that dice don't have a 0. My idea was to equalize the ranges by subtracting 1 from the result of the die roll. Thus, eg, if you rolled a 1, then subtracted 1, your result would be 0. OTOH, if you rolled a 4, then subtracted 1, the result would be 3. Also, the reason I didn't use a 6-sided die is that the coin flips would have required 32 lines to lay out all the possibilities, & I'm just too lazy for that.
In some particular package?
I've updated my response to include McCullach and Nelder reference.
Stock prices simply do not behave like this. It might help if you revealed what these numbers really represent.
I can't see any way to do this, short of a power series involving incomplete gamma functions or something equally unhelpful.
can I assume you know the eigenvalues of $B$? If so, no approximation is needed.
Often, a rule-of-thumb is stated that the VIF should not exceed 5 or 10. These rules have to be adopted [with caution](http://www.nkd-group.com/ghdash/mba555/PDF/VIF%20article.pdf), however. There are other measures of multicollinearity, see [here](http://stats.stackexchange.com/questions/48822/is-there-a-reason-to-prefer-a-specific-measure-of-multicollinearity) and [here](http://stats.stackexchange.com/a/56664/21054).
Does it really make sense to classify on the _name_ of an establishment?
I have found package(aod) which includes functions donner() and raoscott() - which seem appropriate chi-squared modifications. I would certainly appreciate a second opinion on which to use, and what options are appropriate: raoscott(cbind(y,n-y) ~ group, data=matrix)
I'm unclear what "covariance type" means here - what are the options?
I don't know what exactly do you mean by poor prediction, but prediction can be meaningless if you have outliers in your data, for example: construct data using formula y = -10x for x in (1, 2, ..., 20) and add point (1000, 1000) to the data. The corelation is 0.966, but linear regression model is following y = -116 + 1.1x, which is quite far from y = -10x.
Out of the three points you made, only (1) makes sense. A good classifier that generalizes well, will have high recall and high precision. And there is no such thing as a _generalized_ classifier.
**Very** closely related: http://stats.stackexchange.com/questions/30728/how-small-a-quantity-should-be-added-to-x-to-avoid-taking-the-log-of-zero.
Sorry for the late response. I was rather thinking about a general design, but I've never heard of JMP so I'm going to check that out.
This looks a lot like a homework problem, and if so, please add the `homework` or `self-study` tag.
I still have absolutely no idea what this is about. I think you want to create a ranking system as a measure of something, but I don't know what. What difference does it make if it's hard to keep a good ratio w/ a great tracking period?
thank you @user30523 for your reply, I have added more information above!
That could work, what's the benefit over KL?  But if I still wanted to go ahead with KL-Divergence, how would I go about that?
In what sense are the predicted values normally distributed? In general, for a specific point in the covariate space (ie, a specific set of predictor values) the predicted value is a single point--it has no conditional distribution. If your covariates are normally distributed, the *marginal* distribution of the predicted values might be normal, but that's a different issue.
There is no such thing as the random variable $Y|X$; what you are saying is that the _conditional_ density of $Y$ given the value of $X$ is a normal density.
You ought to take a quick look at the StackExchange intro pages and our own [faq], Franklin, because--taking your comment to be sardonic--it sounds like you have a misapprehension about what this site is. If the duplicate does not answer your question, at least having it pointed out that it *does* provide an answer should have advanced your research. I invite you now to modify your question to show where the disconnect is in your mind between the information in the duplicate and the information you are looking for: that may make your question both unique and helpful to you.
At some point--probably several points--you will need to invoke and exploit a definition of conditional probability.  How you proceed depends on what definition(s) you know.  What are they?
No it was a mix (for most of the companies). They selected online for employees having computer access and paper for others.
Duplicates: http://stats.stackexchange.com/questions/6563/80-of-missing-data-in-a-single-variable and http://stats.stackexchange.com/questions/30728/how-small-a-quantity-should-be-added-to-x-to-avoid-taking-the-log-of-zero.  Both treat different aspects of this question, but together they cover it fully.
"*when we try to calculate the error between the estimator $\\hat\\beta$ and the true $β$*" -- as with predictions,  there's two competing effects going on, one helping you and one making things worse; variance is generally improved, bias is made worse. A little shrinkage may be a good thing. A lot of shrinkage may be too much.
Thank you for answering, many voted, but not so many answered =) I have been looking at Multi Hypothesis Tracking for a while now. I will check out your suggestions, have not heard of two of them!
You might get more luck if you posted the data as a `dput` of an *actual* R object; don't make readers put frosting on top as well as bake you a cake!. If you generate the appropriate data frame in R, say `foo`, then edit into the question the output of `dupt(foo)`.
+1 It may be worth noting that in some circles the term "lognormal distribution" may comprise the three-parameter version in which an additive location parameter is included. In this case, the answer--by explicit construction--is *yes*.
Thanks for the very helpful discussion.  Some useful follow-ups -- different definitions of quartiles and hence IQR (John), standard deviation not in fact standardising (Harvey), and QQ plots as a tool to compare two distributions (Peter).  (+1 to all three answers!)
@ils well, i know that R will not allow you to fit this model. I got the numbers from Tabachnick & Fidell, Understanding Multivariate Statistics, and a closer look at these assumptions from MacCallum et al http://notendur.hi.is/adg11/Proffraedi/Fabriger__Wegener__MacCallum_og_Strahan_(1999).pdf
R has the [truncreg](http://cran.r-project.org/web/packages/truncreg/index.html) package.
@Alexis, that an estimate is wrong in a given sample does not imply that the estimator is biased. Consider a case, where we have a single outlier with much high variance: sometimes, this will lead to $\\hat{\\beta}$ being higher than $\\beta$, sometimes lower. However, *on average*, it shouldn't make it differ (thus the unbiasedness).
@whuber Sorry, I edited it. May be you already noticed that in the second last table called "Type 3 analysis of Effects". The Pr>ChiSq for Facility_Or_Organization is <0.001. However, in the last table called "Analysis of Maximum Likelihood Estimate" the Pr>ChiSq for each Fasicility is quite high around 0.96. For example, for first Facility ARKANSAS NUCLEAR, the Pr>ChiSq is 0.9670. I just want to know why the Pr>ChiSq for whole Facility_Or_Organization is quite low but for each of them it is high?