tttjjj/elt_bench huggingface dataset:
What kind of GA? Compared to what "traditional" methods? Without this, one may only say something like "Faster convergence and smaller danger of getting stuck in local optimum in some applications", same as for any other optimization method.
I used 10 for the example; replace that with N where you determine N as the length of your dataset.  Also, this is simply  an application of **weighted least squares** which any decent regression textbook will cover in more detail.
I used the similarity matrix as co-variance matrix inside Gaussian Process.
http://davegiles.blogspot.com/2012/02/minimizing-length-of-confidence.html is an excellent article on shortest length confidence intervals-with a description of the case when the pivotal statistic is assymetric, in the second half of the post.
Why apply non-metric methods to measured data? More generally, it is best to assume ignorance of biology when posting here. People don't necessarily know how biomass is measured.
Crossposted at http://stackoverflow.com/questions/13532997/bootstrapping-standard-normal-data-does-not-produce-a-mean-of-zero-and-standard
gtk is fine on OSX.
It remains uniform then. Thank you!
Thanks for your insight. I think I see what you mean. This may also be connected to the issue of distinct parameters (in his 1976 paper, Rubin shows that the missingness mechanism is ignorable when the missing data are MAR and when $\\phi$ and $\\theta$ are distinct). I'll have to think more about this.
@Aly: Wikipedia, really. (it is the lemma : Tikhonov Regularization). The method that whuber mentioned using the SVD will give you a non-negative definite matrix if you set small eigenvalues to zero; you will still need to add some small constant to all your eigenvalues to make them positive definite. Practically both methods do the same. Just I resorted in not using the SVD but directly affect the samples eigenvalues by adding $\\lambda$ to all of them. I haven't come across any references, both methods are quite intuitive I believe.
+1 nice answer, very thorough.  Forgive me, but I'm going to nitpick on two things.  1) the p-value is understood as the probability of data being as extreme or more extreme as yours under the null, thus your answer is right.  However, when using discrete data like your coin flips, this is inappropriately conservative.  It's best to use what's called the "mid p-value", i.e. 1/2 the probability of data as extreme as yours + the probability of data being more extreme.  An easy discussion of these issues can be found in Agresti (2007) 2.6.3. (cont.)
By standard gaussian I mean the one centered at the origin and with Identity covariance (I will edit this in). Mahalanobis distance wrt to the Identity covariance matrix==Eucledian distances.
Please define the notation.
@user: If $b=0$ then my method fails because of division by $0$, and in practice $n$ becomes an absorbing state.  If both $a=0$ and $b=0$ then you have two absorbing states, at $0$ and $n$, and so there is no unique steady state vector.
See [here](https://www.otexts.org/fpp/8/8) for a description (basically past $e$ values come from residuals, future ones are replaced by their expectation, 0).
what you're looking for is a concept called "statistical power".  google that first.
+1 The extremely pathological case where $X = Z$ highlights this further.  $Y = b_0 + b_1 X + b_2 Z + e$ and $Y = b_0 + (b_1 + b_2) X + 0 Z + e$ would be indistinguishable.
With an ordered outcome you porobaly don't want to do nominal regression.
Glen_b: Have you got a reference for the calculation you proposed (book, article, ...) to dive deeper into that matter?
... so I'd like to avoid this.  Also, many external libraries will make it difficult for me to send the source code to others (read: my boss) and explain how to compile.   So I'd prefer a simple and simple to implement solution (in C++, in Mathematica it's easy to implement).  I'm looking at the recurrence plot now, perhaps I might be able to come up with something simple enough using that.
If the user did not made money, the value is 0. I am assuming that the distribution of the sample average of $\\theta_1$ and $\\theta_2$ are normal based on the [CLT](http://en.wikipedia.org/wiki/Central_limit_theorem).
I undeleted the image because without it, the answers are incomprehensible.
Could you please explain *why* this gives you a cutpoint and what optimality properties it has?  (This is related to @Stephan Kolassa's inquiry about the loss function.)
This question appears to be off-topic because it is about how to do something in SPSS. It should be asked on an SPSS list.
It should be the other way around. If you're assuming a gene-dosage effect you have only one parameter and it's a one degree of freedom test. If you dpn't assume the gene-dosage effect, you have 2 parameters and if you want to test them jointly it's a 2 dof test.
As for _defined_, loadings are - as you know - true correlations between factors and variables. Geometrically, a communality of a variable is a projection of the variable's variance onto the "factor subspace", and the loading is a coordinate of this projection on this or that specific factor. As for _computationally_, loading depends on the method of factor extraction (e.g. principal axes factoring or least residual factoring, etc.)
@schenectady Use $$ for LaTeX in comments, please.
Exact, I was mainly asking whether my measurements are independent random variables or not. Because a measurement is different from tossing a die.
Parts of this question are confusing, because initially it sounds like just one "weighted powder bed" is involved, and therefore the number of particles in it is *known* ("a precise counting of the particles [was] done"). What would be the point of combining the two RSDs then? If multiple powder beds are involved, why not just summarize the counts in each bed? Also, do these data represent a sample of something? What of?
*Sparsity and the Bayesian Perspective* is a good paper that explains well.
Can you also help with part b) and c) please?
Does "linear response" mean that you're dealing with a *continuous* response (or manifest) variable in [0;1]?
If you actually want the random numbers to change every time you start Stata: one idea would be to base you "seed" on your clock using your profile.do file. Try ``set seed `=clock("$S_DATE $S_TIME","DMYhms" )-162e10'``
The MLE would presumably be the $\\lambda$ that will maximize the likelihood, since that's what it's actually for.
In the second graph it looks like the plotted point for the second peak is very close to the actual peak, while in the first graph the closest points to the second peak are on either side of the actual peak. Redo your calculations on a finer grid and they'll probably look more alike.
Yes of course. I suggest you follow @whuber's advice and edit your post to explain exactly what you want to do. You'll get more appropriate answers then.
Hint: You can solve this problem by drawing a figure like the right hand one at http://stats.stackexchange.com/questions/47671/what-is-the-difference-between-using-the-multiplication-rule-or-using-venn-diagr/47800#47800.  (The two dimensions of the rectangle correspond to "Receives offer from A" and "Receives offer from B.")  Your question is *almost* identical to the second half of http://stats.stackexchange.com/questions/47671. There, a conjunctive probability (0.10) is given; here, a conditional probability (0.70) is.
@Gleb_b Ripley's views can be deduced from his 1996 book on pattern recognition, to which he referred to in that R list post, see e.g. chapter 2. The mathematical details and comments on 
other aspects of Akaike's work (e.g. what does A in AIC stands for) make me think he understands this area well and had read more than just a few of Akaike's papers.
Try increasing the  `workspace` argument of `fisher.test`. That may avoid the error message, if your computer has enough memory (and you have enough patience). See `?fisher.test`. See my previous answer http://stats.stackexchange.com/questions/4023/chi-square-test-for-equality-of-distributions-how-many-zeroes-does-it-tolerate/4029#4029
How long it takes to do a 5-fold CV for a single binary classifier?
@chl, could you explain why there is a risk of overfitting by using only 5 features?
OK, thanks for the clarification.  Consider what the difference in $\\frac{d^2}{d\\theta^2}\\log L$ would be between the two likelihoods.
http://stackoverflow.com/questions/4070483/text-retrieval-using-r
For this answer to be practicable, you need to stipulate, in detail, how one "appropriately normalizes" the values and you also need to provide some way to estimate how large $n$ must be before the asymptotic formula becomes a reliable approximation.
Binning data that is continuous and then computing a correlation is like cutting off your leg and then getting crutches.
@JEquihua I looked it up, but it didn't seem to get me anywhere.  Can you tell me if you use a two stage least squares?
@DJE, you might like [this XKCD comic](http://xkcd.com/1138/)
Have you had a look at the [meta analysis task view](http://cran.r-project.org/web/views/MetaAnalysis.html)? Maybe the `metafor` package is what you're looking for?
if there are zeroes in your data, exponential regression might not be appropriate, since the model as you stated it cannot allow zero values to be observed.
Shouldn't it be $(\\mathbf{y} - \\mathbf{b})^\\top$ ?
I know that if $U\\sim Uniform(0,1)$ then $F^{-1}(U)$ has density $f$. Hmm..if$F^{-1}$ is a strictly increasing function then i guess it would map the smallest uniform to the smallest from density f.
Can you assume that the missing data is MAR (in the terminology of Rubin (1976))?
Standard SVM does *not* perform feature selection inherently. The solution of SVM training is sparse in terms of the training instances, e.g. proteins, not their attributes. You can consider L1-regularized linear SVM for feature selection.
$Y-E(Y|X)$ is a variable with zero mean. Hence $var(Y-E(Y|X))=E[Y-E(Y|X)]^2$. Now $Evar(Y|X)=E[E((Y-E(Y|X))^2|X)]=E[Y-E(Y|X)]^2$. A bit less complicated second part of the answer.
Is there a way to get 366.3483 for x = 1100 without using the predict function?
Thank you for the additional details gung!
This question is extraordinarily broad. Unless the scope is narrowed and some more context added, it is very likely to end up closed. Without meaning to sound unnecessarily harsh, it is akin to asking: "What are some examples of flowers that are not red?" (Cue Harry Chapin.)
Also, should there be a index for `regvar[]` (i.e. something inside the brackets?)
@Alexis There are no assumptions in the Pearson correlation coefficient. It's just a measure of the linear association. Assumptions about the distribution of the coefficient only come into play if you want to do hypothesis testing. This kind of hypothesis testing is inapplicable for the situation where you want to test more than two variables, as Ramzan wants.
I might suggest providing more details about what exactly you DO know, as your question is rather vague.  Is it possible that you sample some people who are using services to ask how many (and which) services they are using?
The kind of independence to explore concerns the combination of all three characteristics. E.g., if AX3 is a marker for a syndrome that includes baldness in females (but not in males), then any long-haired person with AX3 is necessarily male, making the probability of being female 0%, not 97.3%.  I hope this makes it obvious that anybody producing a definite answer to this question *must* be making additional assumptions, even if they do not explicitly acknowledge them. The truly useful answers, IMHO, would be those that show directly how different assumptions lead to different results.
You may want to edit your question to explain that it is a time series. (Same dataset as http://stats.stackexchange.com/q/87845/32036, right?) This makes the answer somewhat more complex. It's also rather unclear what exactly you're trying to find out. Is it whether your climate variables affect your moth variables in a given season, differently across seasons, etc.? BTW, are abundance and richness continuous, count, or discrete variables? I assume your climate variables are continuous, but this may matter somewhat less...Information about the distributions' shapes would help too.
You may well find your answer if you search this site for "credible."
Can you give details about your data set, I mean, your dependent variable $y$ and independent variables $x_1$ and $x_2$?
This follows from the fact that the sum of the first $N$ integers is $N(N+1)/2$. This can easily be shown using induction in $N$.
Also of interest is the related discussion of hypothesis testing with large sets of data: http://stats.stackexchange.com/q/2516/919
Do you *know* that these processes are first-order Market chains (and, if so, how)? Assuming the answer to this is affirmative, then what additional *a priori* information do you know about the structure? I'm trying to identify why you are interested in clustering in the first place; I suspect knowing this will help our readers guide you more efficiently to a solution.
Additional question: and if you want to present the z-statistic and p-value, how would you do that ?
@Benjamin: Thanks for the reply. I am not asking for a closed form solution now. If we denote the solution of $\\frac{\\sum_x xe^{\\theta x}}{\\sum_x e^{\\theta x}}=\\frac{1}{n}\\sum_i x_i$ by $\\hat{\\theta_{\\text{ML}}}$, is there any connection between $\\hat{\\theta_{\\text{ML}}}$ and $\\frac{1}{n}\\sum_i x_i$, like, $\\hat{\\theta_{\\text{ML}}}/(\\frac{1}{n}\\sum_i x_i)\\to 1$ as $n\\to \\infty$?
Great answer! Thanks!
Have you considered using propensity scores as a covariate?

I don't know if matching between 3 groups is available; it sounds like it would be very hard.
Have you considered applying  _[Jensen's Inequality](http://en.wikipedia.org/wiki/Jensen%27s_inequality)?_
*"In mixed effects models, the "predicted value" for the true response of the ith subject on the jth trial (Yijˆ) is a weighted average of the response that was actually observed, Yij, and the population averaged mean-response profile, Xiβˆ."* This is not quite right as stated. Do you perhaps mean to say that the predicted subject-level **coefficients** (not the predicted values at the observation level) are a weighted average of the coefficients computed using only that subject's data and the coefficients computed from a simple regression of all of the data?
As the OP pointed out in a comment, "In Jaynes' framework you must always condition on something. P(X) is undefined for all X. Thus conditioning on a single term is unavoidable."  So, you are not allowed to write P(C), P(A), etc.
Well, it could certainly make sense, indeed such a model is relatively easily understood and interpreted; whether it does make sense in your particular application depends on what the application is.
Brilliant. Thanks for the clarification.
Converting factors to continuous value is equally bad for RF as for linear regression -- it may decrease your accuracy but it is unlikely that this will completely spoil the model.
As pointed out above, can you assume independence?
Thank you, this is a great answer; it gives me a good idea about my dataset and further it narrows down the information I need to look review considerably. I guess that what you mean by YOB being a continuous variable is very similar to the question whether age is thought to matter or not for the outcome. Thanks again.
Nice question, I bet it would help to state your sample size to interpret the residuals for the reproduced correlations. In smaller samples we might expect such deviations with more regularity.
r is clearly not normal. Its distribution is right skewed (lot of small values close to 0, few large values). You will see this by typing "hist(r)".
A couple of other questions/remarks: It seems implicitly based on how you are proposing to construct $p_i$ that you are really considering a triangular array $Y_{i,n}$, $i=1,\\ldots,n$ for the purposes of convergence analysis. From how you construct the $p_i$, it seems you should also be able to (just as easily) sample Bernoulli random variables with conditional probability of success $f(U)/c$ where $U$ is a uniform random variable. Is that true? (A little more context to your question would likely resolve a lot of these queries.) Cheers.
Got it now. When you introduce Z you consider Z and X to come from the same distribution.
I do not see how you could possibly do this without making some assumptions about the distribution, for then *any* point configuration is consistent with a likely realization from some 2D distribution and, relative to it, does not "lack" any points in any region. At a minimum you must make some assumptions about homogeneity or stationarity.  What is the practical problem you are really trying to solve?
Use the `Arima` function from the `forecast` package with the `model` argument.
The calculations are explained and illustrated at http://mathematica.stackexchange.com/questions/21396/how-to-use-a-33-covariance-matrix-to-plot-an-error-ellipsoid/21417#21417. A user has noted that this question is cross-posted at http://stackoverflow.com/questions/16659749/getting-the-ellipse-function-to-match-the-dataellipse-function-in-r, where it has been considerably amplified and discussion has begun. Therefore this version will be closed.
That point, $p = (3-\\sqrt{5})/2$ is also the fixed point for iteratated minimax of $n=2$, where you are taking maximum of 2 minimums of 2 draws from $U(0,1)$ (e.g. the solution to $x==(1-(1-x)^n)^n$.  Iterations of minimax increase $F(x)$ (the cdf) for values below $p$, and decrease it for values above $p$.  Any thoughts on what might be the relationship between the two?
How many years of records do you have?
Are you sampling both GP's at the same points of the index set? I mean, are you fixing $t_1,\\dots,t_n$ and observing the values of the random vectors $(X(t_1),\\dots,X(t_n))$ and $(Y(t_1),\\dots,Y(t_n))$?
+1 I worry that lots of resampling will only be confusing and misleading.  Resampling in a controlled way, via cross validation or a hold-out sample for verification, obviously is not problematic.
At least in my case, I'm somewhat in the same boat. I appreciate Mondrian and keep it up to date, but when I actually explore a new dataset it tends to be in R, which is less interactive but more flexible overall. I started writing you a full answer and realized that I was speaking in the theoretical and not from actual experience.
On what basis can you assert "*which would not happen if the scoring is done properly*"?
For reference, `BRT_MODEL$self.statistics$correlation[[1]]` is the correlation of testing to training data, which is a good test metric.
"non-normality isn't an obstacle for r" - Wouldn't you agree that 1) if a distribution is nonnormal because it contains outliers, they can seriously distort r?  2) lack of normality tends to attenuate correlations, and that part of the reason we often transform variables is to see how much stronger an r we can obtain that way?
I've written a short description of what those values are.
+1 Great Solution @cardinal.
By the way, 6/20 is not 33% ;)
Can you give examples of results you expect or even hypotheses? This will influence the answer. For example, I suppose combined change over time in longitude and latitude (i.e. speed) might be interesting. Or only change in altitude over time? On the other hand, change in weight over time maybe not?
Thanks for your comments. I have edited the title and removed my old post that did not ask the proper question.
Not really, no. But just by doing all pair-wise comparisons (between pairs of clusterings) you will obtain a lot of information. Notice that with regular points in 2-D space (or any dimensional space) the usual notion of distance does not extend to triplets of points either. It is also useful to be aware that these distances can be split into two components, each indicating the distance to the greatest common subclustering of just one of the clusterings. This can give you additional insights into subsumption relationships, i.e. "(nearly) being a subclustering of". See also the link above.
@user785099 Try both :)
If your distribution is bimodal, as you indicate, measures like mean and variance can be hard to interpret. For example, large variance could mean the two modes are far from each other or it could mean that each mode, separately, has a lot of variation around it (or both). As a summary statistic, it lacks clear meaning. It may be that the bimodality arises from aggregating two different distributions. If that is the case, you may want to disaggregate (e.g. by using a mixture model) before you look at things like variance.
You need to provide more information on your dataset to get a reasonable response. Also, when you talk about 3-10 clusters, is this the result of, say, eye-balling a simple plot of the data?
Thank you, Corey! I wonder if there is an example showing the use impacts() function in the package. Are you aware of any?
Sure. I'll add that in case of randomForest, OOB votes are in `votes` element of the `randomForest` object; for prediction, one must use `predict` with `type="votes"` or `type="prob"`.
The formula is for the population variance, whilst Matlab probably computes the sample variance (with factor $1/(n-1)$).
At any rate, I'v upticked you answer, as amended. It both answered my original question, and (more usefully if more frustratingly) persuaded me that it was not the question I wanted to ask.
You should not cross-post. Reading the FAQ would have let you know what's on topic here *and* not to cross-post. If you wanted this moved there, you would flag it and a moderator would shift it. Likely one of the two will be deleted now ... probably this one. If I am lucky, it will get moved and then combined.
It seems like you already had a reasonable answer on the other site?
...of course, if anyone knew of a link to the lectures that would be very useful...
(+1) Stas, is there something fundamentally different here from *quota sampling*? Just curious, a cursory reading of the question makes it look like just a change in terminology. Also, briefly, what are the main critiques of the Iraq cluster sampling study? I remember seeing this when it first came out and reading a bit on it.
question: You say "I know how to interpret the above." but the title of the question states otherwise. What is correct here ? Up to this point I answered only the questions appearing in the body.
@Stefan, It is still totally unclear what you want, or why this needs to be accomplished in SPSS as opposed to whatever language. Feel free to clarify your question if you are in search of something different.
Thanks.  If I go down the preventing overfitting road there is potentially a lot more work to do.  I guess I should just present both solutions and they can pick the one they like :)
Can you clarify what the "certain task" is that you want the SVM to do?
@alittleboy. Are the findings (on average) in fact in the direction opposite to your hypothesis? You should be able to figure that out by looking at the differences, without thinking about P values.
@ssdecontrol One example is when the parameter estimates arise from a single regression; in some circumstances it may arise when there's dependence between observations in several regressions.
We generally put the response on the vertical axis.  Visual inspection suggest that even if there is a real correlation, variation in entry point quality makes it pretty useless.

But since you're dealing with stock prices, things get even more complex because time series are involved.
@Scott, if your hypothesized proportions for the three groups are that they are all equal then the expected value should be 1/3 for each group.
Why are you looking for a heavy-tailed distribution? Perhaps some context would help getting an answer. Your question might be a bit too broad as it stands now.
http://en.wikipedia.org/wiki/Ratio_distribution.  For a more specific discussion, please see http://stats.stackexchange.com/a/3644/919, which answers your question (because $(a_j, A)$ have a bivariate normal distribution).