tttjjj/elt_bench huggingface dataset:
Hi, actually my other account was unregistered and i couldnt access the question when i got to my computer, so i tried to register the account by making an account with the same name and it didnt work. Also im new to this site and after looking at other questions i realised that mine was more yahoo answers standard and not very clear so i wanted to edit it (not that its much clearer now), but still couldnt so re wrote it here.
You could try to generate raw data corresponding to your matrix of euclidean distances and input those to K-Means. Alternative easy approach could be to use Ward method of hierarchical clustering of the matrix: K-Means and Ward share similar ideology of what a cluster is.
@RobHyndman updated data
+1 This excellent analysis demonstrates why the question is appropriate on this site.
Could you add some addiitional information. (1) How many factors do you have? (2) How many observed variables per factor do you have? (3) What is your sample size? (4) Are you including the second order factor in these models? (5) You sometimes talk about two models, and other times three models:  (a) factors uncorrelated; (b) factors correlated equally; (c) factors free to correlate, how does that relate to problem 1?  (6) Beyond explaining your problems, could you add one or more clear questions at the end?
Your last sentence made me curious, so I tried this out with your example. I used the ``boot`` package and had it provide all the possible intervals: the first order normal approximation interval, the basic bootstrap interval, the studentized bootstrap interval, the bootstrap percentile interval (which your code already provides), and the adjusted bootstrap percentile (BCa) interval. Results were essentially all the same (rejection rates around .16-.17), except for the studentized interval, which had a nominal (.05) rejection rate (which makes sense, since it is more like the standard t-test).
@user27525 Yes, it does. You don't even need a kernel. You could just use the dotproduct and you'd always be able to separate them.
Great point about the importance of causal mechanisms.
Related question: [Should I reverse score items before running reliability analyses (item-total correlation) and factor analysis?](http://stats.stackexchange.com/q/2863/930)
Your question seems rather vague. There may or may not be issues, depending on the circumstances.
To obtain the estimator one needs to take the partial derivatives 
$$\\frac{\\partial J}{\\partial \\mathbf{A}} = 0, \\qquad \\frac{\\partial J}{\\partial \\mathbf{b}} = 0$$. The resulting linear estimator has the form
$$\\mathbf{\\hat{x}}(\\mathbf{z}) = \\bar{\\mathbf{x}} + \\mathbf{P}_{xz}\\mathbf{P}^{-1}_z(\\mathbf{z} - \\bar{\\mathbf{z}}) $$
@BigDataLover [Semi-supervised learning](http://en.wikipedia.org/wiki/Semi-supervised_learning) might be worth a look to include the customers which did not respond into the model. I second this answer (+1). I think it is better then your solution because building two binary models instead of one trinary model will lead to an increased error since you have to perform a classification decision in between.
I'm a bit confused about why your questions are about the global mean and standard deviation but your main research question is about the difference between different samples.  Can you clarify that this is correct, and if so, say why you need the global mean and standard deviation?
Holding $x_2$ fixed, yes. But the sign of the effect depends upon the sign of both $\\beta_1$ and $\\beta_2$. And you really aren't holding the quadratic effect constant, you're holding $x_1$ constant.
This is a version of what is called a deconvolution problem: if you move to the log of the product, you get the estimated distribution of the sum when you know the distribution of one of the terms. Check on [wikipedia](http://en.wikipedia.org/wiki/Deconvolution).
Are you using the libraries from a host application (like R) or from the commandline (or from your own application)?
Indeed that was the problem. There were several categories present in these data that had only 1 subject endorse them (due to some prior data manipulation), and therefore the intercepts were not identified.
At least  you should precise how many observations and how many covariates you have. And is it a real dataset ? Do you have some reason to expect that the response is well associated to the covariates ?
I understand Rubin's point to be that you can get an unbiased point estimate using careful single imputation, but the standard errors will be wrong. However, in many cases mean imputation will have other problems, in particular distorting patterns of association with other variables.
Does the mean of a random walk exist?
@gung That wasn't intended as criticism in any sense; just trying to make sure Hibernating was aware that a reference to the original paper was there, since that's what was asked for.
How many tags do you have?
I know that I should go further to get accurate estimates, but I was hoping to understand first the GQ as a first step.
Thanks.  Which methods would you recommend I research to try and replicate that?
+1. This is one of the techniques recommended by the comp.graphics.algorithms FAQ ["Uniform random points on sphere".](http://www.cgafaq.info/wiki/Uniform_random_points_on_sphere)
Hello Mike. Thank you for the answer, and for your package - it is really great!
Welcome to crossvalidated. I'm not sure whether this question is better suited for cross-validated or stackoverflow (where programming questions are discussed). IMHO it is more a programming than a statistics question , so I'll flag it for moderator attention, and the moderators can decide whether it should stay here or be moved to stackoverflow.
[From wikipedia](http://en.wikipedia.org/wiki/Harmonic_mean#Harmonic_mean_of_two_numbers): `Typically, it is appropriate for situations when the average of rates is desired`. This thread might also be of interest: ["Which “mean” to use and when?"](http://stats.stackexchange.com/q/23117/10525)
Isn't that for independent groups, not repeated measures?
"If the original sample on which the bootstrap distribution (and hence all of the results that follow from it) is not adequately accurate, then your results (and hence your decision based upon those results) will be flawed." -> but does the bootstrap CI perform worse than the analytic alternative in these cases?
I don't need to see/know/understand source code of the platform I'm using...
Unless the index is some *meaningful quantity* (e.g., directly related to a time variable), this plot doesn't tell you very much. You might start by looking at a plot of the residuals versus the fitted values.
I previously had a [similar question](http://stats.stackexchange.com/q/3650/1036), and the [hypergeometric distribution](http://en.wikipedia.org/wiki/Hypergeometric_distribution) is applicable to your question as stated.
How about some kind of iteratively reweighted least-squares regression, in which you put increasing weights on the observations whose fitted values have incorrect sign?  Look up "boosting".
Hmm, I have to say that then, I dont get it. Can you clarify in a little less mathematical terms, what this constraint means?
@Scortchi agreed, although there is a great deal of correspondence between the two distinctions: observational data, error prone and subject the infinite unmeasured mediation pathways, is carefully adjusted by measured confounders, but the results are important to interpret as "population averaged" which is why large public health data almost *necessitates* GEE (the empirical approach). On the other hand, experimental *in vitro* data are so carefully controlled, little wonder analysts favor measuring and fitting sophisticated probabilistic models for their data (the mechanistic approach).
"*My approach is rather naive* ... *why the weather forecast could be done so great?*". Simply: because it's not so naive.
What would the point of the similarity score be? For example, it's entirely sure that "Sun" is a large IT corporation, just as Microsoft is, but it's surely likely to have a much higher score for "celestial object", so your similarity may be quite low...
Could you clarify what you mean "which voxels are correlating"? It sounds as if you just want to identify sets of voxels having common values.  (If this is a one-bit image, there are only 8 possible common values anyway.)
Dan Carr's micro maps take along time to digest the full graphic, so does any detailed road map. Neither is necessarily a bad thing. See James Chesire's blog post, [Fast Thinking and Slow Thinking Visualization](http://spatialanalysis.co.uk/2012/03/fast-thinking-slow-thinking-visualisation/).
Hurlbert, S.H. (1984) Pseudoreplication and the design of ecological field experiments. Ecological Monographs, 54, 187–211.
@user19856 has suggested the following "minor characteristicos implementation": There are some "variations" of the same method: some "fix" one vertice on centrer of picture and let the other "to gravite" around it. I dont know how these implementatins works, but if you look the code, perhaps you can fix some vertice (e.g. the most connected) AADF
(+1) Good point.
For anyone interested, I found a way around this. I decomposed each degree by clustering the specific subjects they have to take and used those subjects as outcome variables. So instead of taking a general GPA as outcome variable, I ran Hierarchical Regression on specific subjects.
Thanks chl.  I guess it is something to start with...
At least Hamilton expresses it like this... Which then results in the conclusion that the variance degenerates to zero, as the part that converges to a functional expressed by a brownian motion has a finite variance. My point is however not to question Hamiltonian semantics, but how to arrive to the convergence in probability to a mean of zero
Hi Whuber, thanks for your reply. I don't have matlab as of yet (which if I've had, could be a lot easier I suppose). Can we do the estimation in Eviews? Or could there be other options to estimate these parameters?
Thanks @peter-flom, the model in stata is quite similar to R. No idea what the SAS equivalent is. I don't use it. In `family(binomial total_tests)` the `total_tests` is the number of Bernoulli trials.
Don't just remove observations because it makes the data "more normal"! If these outliers are valid data points (i.e. the measurements are okay), you have to include them in your analysis. As Nick Cox said: you might try to transform the data. A non-parametric test like the Wilcoxon test might be usefull as well.
How do you think about the problems within? I happen to own this and want to do some exercises for enhancement.
Randomness is good, but is it really necessary?  This situation appears to be akin to a scientific experiment in which the investigator *purposely* sets the dial of a black box to a determined value (analogous to employment status) and observes an outcome (the dependent variable).  All we really need is that the *residuals* in the regression be independent of the selection of the individuals.  Thus, even if this dataset were not created by randomization, it still could be rescued.
Why need an "agreement about what features such a rubric ought to include?" For a rubric to be successful, it should be project-specific and better if the students were also involved in creating/revising the rubric. The cookie cutter one just wouldn't cut it (pun intended.) Base the items of the rubric on the learning objectives of the work (you have some, right?), for each expectation, list out a spectrum of description from outstanding to aspiring (try not to use negative word.) Pass it around the students for comments, make sure they feel that the top levels are attainable. That should do.
Is he looking to perform statistical analyses, interpret the output or critique published papers that use statistical methods?
I found that there's a package in R called "nnls", which stands for non-negative least squares which solves this problem using the method you've described. It's a pretty old package.
I think this is a major improvement. The question is now very clear. Right now I don't have time to give a well-organized response but I'll post an answer later.
And the other problem with the 2 step (first penalized, second not) procedure is that the resulting model's predictive discrimination and regression coefficients will be overstated.  That is because the shrinkage from the first step was not inherited properly in the second.
Is it really the case that a small handful of observations is getting weighted more heavily if the weight is the number of counts of the observation?  From your description, it seems to me that each physical observation is really $n_i$ actual observations, in which case each actual observation is getting a weight of one (using your weighting scheme.)
@Michael: Given the right regularity conditions, the maximum-likelihood estimator is consistent; I don't know under what conditions the bias is $O(n^{-1})$. A quick look at Lehmann gave me "typically" too.
Is your goal essentially to be able to sort your records by P(category = 1), & you don't want any ties?
@DilipSarwate Perhaps simplification was the wrong word? This was self-study, so I will outline the remaining steps.  Combine $P(A \\mid C)$ with $P(C)$.  Then divide numerator and denominator by $P(A)$
Some examples of entering manual contrasts for use with `Anova` and `lm` were given on a related thread: http://stats.stackexchange.com/a/4549/930.
This question is too vague to be answerable in its current form. Can you add more information about your model? You may want to read our [FAQ](http://stats.stackexchange.com/faq). If the question cannot be specified to the point where it is answerable, it will need to be closed. Note also that there are a number of questions already on this site that explore reasons *in general* why reg models might be non-significant; you may want to search the site & read some of the existing answers as well.
@user26008 I clarified and added some comments.
I read in [Handbook of Probability](http://www.amazon.com/Handbook-Probability-Applications-Tamas-Rudas/dp/1412927145) that if the population is infinite, then sampling is usually done without replacement.  When the population is finite, sampling is done with replacement.
Already asked, answered and answer accepted at http://stats.stackexchange.com/questions/67243/determining-an-optimum-and-distribution-range-for-a-given-species-abundance-vs-a
The power depends on the shape of the true distribution of the pairwise differences. You have to specify your assumption what kind of distribution this is.
@dsol: I agree. You could easily be caught out with a "+" in an email address. For a quick and dirty solution, I would probably just skip any punctuation characters that I hadn't specified in my look-up table.
Well, just thinking about the definition of VC-dimension, do you think (say) boosted decision stumps can shatter a larger number of points than a single decision stump?
I'm not sure I would characterize ESL as a reference text. It seems more of an overview to me, i.e., you aren't going to learn the nitty gritty details of (hardly) anything. You will see the broad techniques and overarching themes.
"*should the equation have an intercept term?*" --- yes, definitely.
Well i am not sure if i understand correctly what you are asking but i will give it a try. I understand that you have data and you split them in A and B. You want to know if you train with A what will happen if you test with B?
@zgall1 Thanks for your feedback, I appreciate it. Hm, yes, the transformations didn't seem to have helped much :). At this point, I would probabily try to use splines for the predictors using generalized additive models (GAMs) with the `mgcv` package and `gam`. If that doesn't help, I'm at my wit's end I'm afraid. There are people here that are far more experienced than me and maybe they can give you further advice. I am also not knowledgeable with baseball. Maybe there is a more logical model that makes sense with these data.
Did you look into Manova?  https://en.wikipedia.org/wiki/MANOVA    To get some more specific advice you will need to tell us more about the applied problem you want to solve.  In most cases, with more specific hypothesis, one can do better than a general approach such as manova.
Hi Gung, I'd love to but can't find the time. I am heavily invested with family, work and improving my modelling skills.I am just beginning to work with MARS modelling and am not sure if this will satisfy the same desired outcome as the described cluster + LR ensemble.
Duplicate of many questions [e.g.](http://stats.stackexchange.com/questions/46019/why-squared-residuals-instead-of-absolute-residuals-in-ols-estimation). Also see [this one](http://stats.stackexchange.com/questions/118/why-square-the-difference-instead-of-taking-the-absolute-value-in-standard-devia?lq=1). Why is that tagged `econometrics`? Why did you post this when you posted essentially the same question about ten minutes earlier?
The goal is to find the statistical significance of a certain metric. Lots of data is involved, so best to come up with an efficient method. I started looking into MAD, because of possible outliers due to data problems ...
Welcome to the site, @pythOnometrist. I suspect this is a helpful contribution. Would you mind giving a brief summary of Rpy2, so readers can decide if it's what they're looking for?
So, what you are saying is that AIC doesn't sufficiently punish models for parameters so using it as a criteria may lead to overparametrization.  You recommend the use of AICc instead.  To put this back in the context of my initial question, since BIC already is more stringent than AIC is there a reason to use AICc over BIC?
+1, thanks for pointing out the NADA package.  I noticed it makes it easier to handle left-censored data through the survival package -- is left-censored a common scenario with environmental data?
Hi Akrist. Could you put up a plot of the call series at quarter hourly intervals so that we can see what it looks like, and what might be appropriate?
@Hoser, no it is not a mixture, assuming this is the same question referred to in your other post. Each class is a **2-dimensional Gaussian distribution** with means at (0,0) and (1,3).
@mbq: The LHS must be $$\\log\\left(\\frac{N_0-N_e}{N_0}\\right)$$
Why do you need 8 rather than say 6 digits? Why do you need any digits of imprecision? P.S. There is a special tag called `homework` for homework questions.
Bloody briliant memory. I've 4GB.
A very well-argued response.  For a similar but more detailed one, I recommend Geoffrey Keppel's Design and Analysis:  A Researcher's Handbook.  I'll find references to specific page no.s (1991 edition) if anyone would like that.
On what basis can you assert the distribution of one of the populations?
I mean that confidence intervals always have an interpretation no matter what the statistical test says.  And unlike large $P$-values, confidence intervals can provide evidence in favor of the null.
I tend to follow this approach in practice.  While there is no sampling error, there is still the possibility that some other noise ("luck" or "random chance") could have caused the downturn, and that that "luck" may change in the future.  Of course it may not be truley a random stochastic process, but with no explanatory variables it might behave like one.
@DilipSarwate: Oh, dear. No, I would certainly never "delete" the content of another user's post! Rather, the effort you put into trying to understand the OP's question (and respond to it!) in its original (rather muddled) form is quite Herculean. Rather, I will edit to add a disclaimer at the top so that it isn't inadvertently downvoted!
Thanks a lot for the suggestion, this is exactly what I was looking for! So I would do sqrt(sum((predicted_i - actual_i)^2)/#samples), where i=1..#samples?
(1) What do you mean by "CV"? (2) The first function does not involve the maximum of anything, despite your description of it, nor are its values capped at $2$.  Is there perhaps a typographical error in it?
This looks like a homework problem, which would explain why raw data aren't available and why the problem seems narrowly framed.  If this were a real data analysis, I would (1) probably include the additional ">180" empty bin; (2) figure that it was unlikely to make much difference because it's a small fraction of the data; (3) worry about computing the mean of the exponential from the midpoints of the bins. If it were a homework problem, I would ask the instructor for clarification.
Michael, In the future, can you use the formatting options? (the correct key to insert links is ctrl-l, not ctrl-c).
Thanks again! Regarding integrating feature selection inside the classifier (you mean wrappers?), my understanding is that filter(IG, Chisq...) methods are the simplest to implement and the most scalable, thus they are appropriate to treat very large feature spaces like we do in text classification...However, sometimes using pre-processing step with a filter to reduce dimensionality of the feature space, and then a wrapper yields good results. Anyway, I guess I will stick to filters by now...
@Aksakal: From what I understand, measuring the true values X is actually quite expensive, since it is being done in extreme environmental Situations at remote locations. At the time only 17 measurements were done and it is not that simple to make more measurements.
I would like to see the most basic thing, the fitted line runing through a scatter plot of the data.
It's hard to guess the intent without a context. Your guess might well be correct.
+1 (from earlier). I really like the way `codebook()` lays this out. 1 issue is that `na`s are dropped, which you may want included in your output. 1 way to deal w/ this (at least w/ factors) is to use [?recode.is.na](http://rss.acs.unt.edu/Rdoc/library/epicalc/html/recode.html) 1st (eg, to "missing"); for numeric variables, you can create a new variable immediately to the left of the column w/ a logical value based on `is.na()`, then run `codebook()`. It is a bit of a kluge, though.
**Related**: [In non-negative matrix factorization, does the first N eigenvector have N greatest variance?](http://stats.stackexchange.com/q/43110/2970)
Wikipedia was the first source I referred to, but I started reading about Bayesian statistics and it just went over my head. Thanks for clarifying.
@James  Good points (+1).  I just want to clarify a possible misunderstanding concerning an earlier comment of mine.  In your example *there is no single underlying distribution:* if there is a trend over time, then *a fortiori* the distribution is changing.  When the distribution does *not* change in any way over time, when there is serial independence of the observations, and the distribution is continuous, then the answer to the Secretary problem remains the same no matter what shape this distribution has.
Does it seem a reasonable approach?
I know that this is the general method, but how are D and R estimated, what equations are used for the various structures? What are good initial values? I will check out the pdf now, cheers.
@noname: (cont.)  What you are asking is to show that the problem of minimizing $\\sum_{i,j} w_{ij}(X_{ij} - A_{ij})^2$ with respect to $A$ constrained to have low rank $q$. is not reducible to an eigenvector problem. I am afraid you would need another forum for that (maybe mathoverflow?). But note that finding eigenvectors is also not exactly an *analytical* solution: it's just that the [iterations](http://en.wikipedia.org/wiki/Power_iteration) are usually silently performed by a standard library function.
@whuber: I suspect the point of these problems is to learn problem solving techniques, rather than consume a lot of computational cycles. Symmetry, parity, etc, are classic techniques from Larson's book on problem solving.