tttjjj/elt_bench huggingface dataset:
An SVM for classification is trying to partition the data into two subsets, so you need one label for each of the subsets. Traditionally the labels are $\\pm 1$. This is for algebraic niceness - it allows you to express two constraints (that the margin doesn't contain data points of either class) as a single equation.
First of all, the X is a constant so it will always have a std of 0 (even when X is define as the average of all the exams, X is still deterministic).  X is arbitrary and can be whatever you want and will not change your results (at least of the ranks, the % that one is better than another will change).  I like the mean (or median) because you will get some values about X and some below.  If you set x to 100 all average are going to be below (or equal to) X.  The normal distribution has both positive and negative values.
@Memming: I just revised it to give more context to the questions
@Elvis No offense taken. Though R.A. Fisher should count himself lucky for never having to work with surveillance data collected for other purposes. It's either messy data, or a clinician's gut instinct - I'll take what data I can get.
If time series problems like endogeneity, unit roots and etc are not an issue, i.e. you have repeated measures data, I would try to use something like  `glmer` from package **lme4**.
Certainly the general category of model/test is determined by what the question of interest is, but I think data type definitely comes into it -- after all, if the data were not proportions or counts, one wouldn't even *consider* a proportions test. The data type, therefore, must be at least part of the basis on which such selections are made.
Are you aware of the [academia](http://academia.stackexchange.com/) site? This question half-belongs there, except that there may be three people cross-active on both sites... all of whom have probably replied already :)
If you think that your question was answered to your satisfaction, could accept the corresponding answer?
If this approach is aready being used, then I guess it's not so bad.  Are there other common approaches?
I tested/re-tested/tested-again the formula (line 3, where `rt()` is used) and I'm not sure what the problem is.  Can you be more specific?   Are you talking about a different line in the code?   I've been digging around on the net, and apparently simulating with a t-distribution is not as common as I originally expected.
Hi John.  I believe he might want it on a bootstrap sample.
Is this also correct if did the following: P(disease|+) = P(+|disease)*P(disease)/P(+). 

Where P(+)= P((+|disease) *P(disease)) + (P(+|¬disease)*P(¬disease))  

I get the following values: 0.98*0.008/((0.98*0.008)+(0.02*0.992)) = 0.283. Is this way also correct?
You've got an interesting question. What I strongly suggest you is to give or show your data, for example on 3 scatterplots (v1-v2, v1-v3, v2-v3). Also, tell us please what clustering method and classification method you used and what it was that was `not very good` with their results.
The NIST is usually authoritative, but here it is technically incorrect (and ungrammatical to boot): having a probability defined at "an infinite number of points" does not imply the "probability at a single point is always zero."  Of course they're just dodging a distraction about infinite cardinalities, but the reasoning here is misleading.  It would be better for them just to omit the first sentence in the quotation.
Thanks for the answer guys. Unfortunately, the writing is very vague and does not say much other than keeping the prior explicit, which is confusing and frustrating.
@Dnaiel, tell me if I'm misunderstanding your question, but yes even with symmetric dirichlet prior, say $Dirichlet(0.2, 0.2, 0.2, 0.2)$, draws from that distribution will tend to produce sparse $\\theta$ vectors. By sparse i mean if you were to plot the vector $\\theta$ as a histogram it would be very peaky, rather than flat. In the model above the Dirichlet paramaters are not symmetric due to the $\\beta$ paramater being drawn from a dirichlet hyperprior.
Yes, just these 60. But I think for random forests, each decision tree is created with a subset of the samples, so you can get a estimated generalization error by applying each of the existing 60 samples to only those trees within the forest that didn't see that sample during construction. (http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#ooberr) So maybe typical cross validation isn't needed here.
@rocinante, please make contributions like this *comments*, rather than answers, or develop your points more thoroughly & don't include questions for clarification from the OP. W/ >50 reputation, you have the privilege to comment anywhere.
Thank you, it will be useful.
This belongs on quant.stackexchange.com.
Also see LDA = Longitudinal Data Analysis
@shaw2thefloor ANOVA does not correctly capture the contracted variance implied by a blocked design (unless you mean repeated measures ANOVA?).
According to the abstract, this paper seems to deal with an "error-prone binary covariate": that is, with misclassified *independent* variables only.
(*This a suggested edit by [@Indolering](http://stats.stackexchange.com/users/1689/indolering) turned as a comment.*) I would actually like to just add a comment, but I lack the reputation points. I wanted to point people to [Carnegie's Natural Programming](http://www.cs.cmu.edu/~NatProg/) department, which works on ameliorating many of the problems raised in the spreadsheet addiction link. We are working hard on making programming more accessible and reducing error rates of end-users. It's tangential, which is why I think a comment would be more appropriate than an edit for this post.
There's a vectorized rdirichlet function in R already - in fact several of them (in `gtools`, `MCMCpack` and `dirmult` for example).
But what is the question precisely? Can you give more details?
That's very clever, but not quite what I had in mind! The answer I'm looking for does not necessarily have to be written using the natural parametrization, but the parametrization should be equivalent to the natural one. Your proposal is equivalent to using the natural parametrization with a restricted parameter space, and as I wrote in my question that will indeed result in the cdf losing the proper limits. That being said, I may well reward you the bounty if not other answers turn up!
If your response is a proportion, for logistic regression to make sense you also need to specify what number of cases each line represents.  See http://stats.stackexchange.com/questions/26762/how-to-do-logistic-regression-in-r-when-outcome-is-fractional.   Otherwise you should use a quasi-binomial family for your glm.
I would change "data(:, i) = reshape (image, m*n, 1);" to "data(:, i) = image(:);".  As long as the images are the same size, you might be in business.  what is this class subloop-superloop thing?  It might be your problem.
Not to mention not labeling them at all http://xkcd.com/833/
Good point. I totally forgot about polychoric corr when I was answering. It is the 3rd possibility. One inconvenience, though, that factor scores will not be able to compute because the correlation matrix looses its direct relation to the initial data.
You can average the [Fisher transformations](http://en.wikipedia.org/wiki/Fisher_transformation) of r-scores, if the N's are the same, or perform a weighted average, if they are not. But, I suspect you'll want to take the temporal aspects of this situation into account, if you want an accurate answer.
I agree; with the trivial change that the line fitted can be a curve or surface. Parameters are estimated, models are fitted, but often by the same calculation, e.g. command or function call. "Fitted" and "predicted" overlap in usage. Fitted and predicted values are usually for a response (outcome, dependent) variable.  "Predicted" is not confined to something about the future. However, your context is time series, and time series have their own terminology, which sometimes differs subtly from the rest of statistics. I don't think there is much talk of "forecasting" elsewhere.
Thanks @user1615070; fixed.
(+1) Welcome to the site, James. If you are the James Joyce I'm thinking of (not the Irish novelist), it's very nice to have you here. I have even seen trained statisticians make this mistake on several occasions, particularly when applying the bootstrap or some variation of crossvalidation. Cheers.
Great; In that case you can use panel cointegration test, but it ultimately rests upon whether it is common in your area. But, you can always proceed with fixed effect.
When you say, a "distribution is symmetric", you have to specify with respect to what point is symmetric. In the case of the normal distribution you present, the symmetry is given around $\\mu$. In this case $X-\\mu$ and $-(X-\\mu)$ have the same distribution. In terms of the density this can be expressed as: $f$ is symmetric about $\\mu$ if $f(\\mu-x)=f(\\mu+x)$. BTW, it is good manners to accept answers when you are satisfied with one of them.
It might be worth remarking that "rises to some point and then falls to a steady level without a consequent rise" is distinctly *non-cubic* behavior.  Cubics don't have horizontal asymptotes.  More (quantitative) details would be helpful.
Ok, my formulation was awkward. What I wanted to say is, that the dependence is nonlinear and we could assume something like $E(x_i|Y) = \\sqrt{x_i}$. I hope its clear now?!
Thanks for the answer; I have added a line about my problem. It is basically about Wi-Fi strengths observed at different locations.
@AdamO What type of graph of the data would you like to see? A scatterplot? If not, tell me what would be helpful and I will add it.
Why do you think normally distributed data to use a Pearson correlation?
Note that in this case, a quite weak hypothesis leads to the conclusion that no mean (whence variance) for $1/X$ will exist, i.e., that the approximation in the answer will be rather misleading. :-) An example hypothesis is that $X$ has a density $f$ that is continuous in an interval around zero and such that $f(0) \\neq 0$. The result then follows because the density will be bounded away from zero on some interval $[-\\epsilon,\\epsilon]$. The hypothesis just given is not the weakest possible, of course.
These look like some good examples. Cheers.
If you response is something like case of disease (or not a case) I woudn't call that case and control.
user46788: The coefficient of determination ($R^2$) doesn't necessarily increase on the *test* set when you include more predictors. See @Erik's comment.
Why do you need to check the distributions of the *independent* variables?  Furthermore, because you have separated your 13 variables this way into one dependent one and the rest dependent, one would think that you need to regress the DV on the IVs and check the normality of the *residuals.*
PS: This was posted on mathoverflow.net sometime back. See: http://mathoverflow.net/questions/18302/likelihood-function-for-sequential-random-variables
How do you use structured SVM? The problem I want to solve is standard classification problem. (though the set of class labels has some structure.)
@Procrastinator Good answer, if you put it in form of answer I will give +1 for sure.
Patrick didn't ask the question, he just edited it so addressing him in your answer was a bit misdirected. The OP is C_S.
@pdv yes, the partial dependence can also be done on categorical variables. It must be done carefully though, I re-encourage you to read about it before using it seriously. Also, answers have their own space under them for comments. Please use this and definitely listen to gungs advice.
(CONTD)...But if this were the case, then assume that points 1 and 2 are in the same group: in order to determine whether point 3 should join them or be placed in a separate group, we must measure both $d_{13}$ and $d_{23}$. So we arrive at the internal consistency problem: if for each point we must measure the distance from all its predecessors, then indeterminacies may arise (case in previous comment). If we are to measure the distance only from its predecessor, then we cannot apply the criterion stated for joining a group. What are your thoughts on that?
@ Tal i used to write my code in texnic center (with no highlighting), then call sweaeve(file.choose()), rinse, repeat. Now I just use emacs, write my document in one buffer, have R open in another to test, and then call sweave from the .Rnw buffer and then alter my sweave file based on what happened (any errors or unwanted results). The major advantage to emacs is that R is running right there, so you can replicate the query workflow common in the R GUI, and then copy over the working code to a sweave file. Then, with Latex do all the formatting (again with emacs) and you're done.
+1: Answers the question: "If you use cross-validation to estimate the hyperparameters of a model (the αs) and then use those hyper-parameters to fit a model to the whole dataset, then that is fine…"
However if it is a good choice for monte carlo, it will solved to a required degree of accuracy in short time (with monte carlo). Right?
Mentioning "relible" simply nods towards "representative sample" theme. Mathematically one has error-free regression line with 2 variables and 2 points (individuals). But what if one of the latter two is outlier. No, let us have as many individuals as we can collect, to balance atypical deviates. With your correlation matrix, you just will want to be sure that high correlations in it are significant (reliable).
I believe there's a simple $O(k)$ algorithm, because when $k \\gt 1$, a variance-maximizing subset must consist of the $k_0$ smallest and $k-k_1$ largest elements, whence a search over $k_0=1,2,\\ldots,k-1$ does the trick.
I had not heard of stochastic dominance before but it seems like a good concept to apply to these situations.
+! For windows support.
Nope, that's the whole issue: the distribution won't even be the chi-square. It can be characterized, but is very difficult, and depends on auxiliary parameters. $\\alpha=1$ is the same as $\\alpha=0$, you just rename the parameters.
Thanks for the great answer and for book advice. Also, do you know about relation of window function to generalized functions? It seems (from wikipedia article) they are suitable as domains for functionals.
Apply the following function to your last layer's neurons: 
f_i(x) = exp(x_i-shift) / sum_j exp(x_j-shift), where shift = max_i x_i.

Here f_i(x) is your i-th neuron's output, and x_i, x_j are your neurons, i.e. loop over all neurons x1, x2, ... xi
@mpiktas - I don't think your comment is on-point.  levon9 never said he wants to create regressions without data.  He's looking for macros or code that constitute shortcuts to doing the work of ranking according to some criterion.  This may be "trivial" in the sense of straightforward, but as I see it he's looking for a way to shorten the process.
Because this is a self-study problem, your question should have the `self-study` tag. See http://stats.stackexchange.com/tags/self-study/info
Yes, that's correct. It's not clear why you are puzzled on this, but transformations must be inverted to get back to the original scale.
I suspect this question is answered many times over in the threads you will find by [searching our site](http://stats.stackexchange.com/search?q=%2Bdummy+%2Bfactor)
$s_1$ and $s_2$ are generally correlated, but assume they have unconditional mean zero and variance one. If they weren't correlated, it'd be easy I think...
@biostat, [ESL](http://www-stat.stanford.edu/~tibs/ElemStatLearn/), but I believe that the original (written) reference to this rule-of-thumb is the Breiman et al. book I refer to in the answer.
@Paul The [covariance matrix](http://en.wikipedia.org/wiki/Covariance_matrix#Which_matrices_are_covariance_matrices.3F) of any set of real variables must be positive definite (or semidefinite, if the set is singular).
You could, perhaps with some profit, characterize your data differently: you have a two-way table giving the counts of $(X_1,X_2)$ equal to $(0,0)$, $(1,0)$, $(0,1)$, and $(1,1)$.  The first count is much larger than the rest and the last is much smaller--but so what?  There does not seem to be anything special about this table, so why not apply an appropriate measure of association for two-way tables?
Two last options don't make much sense. The 1st one (replace by the within-respondent mean/median/mode) may be justified if all the questions are supposed to represent the same construct (factor, trait). Better way is to do imputation after you combine the 2 current countries or all 3 countries in one dataset. Multiple imputation is the most advanced choice. But EM single imputation is good too.
More directly, income changes are often multiplicative, at least to a good approximation, as when people in a firm might get a 5% raise. That's multiplying by 1.05, which is adding a constant on any logarithmic scale.
(CONT'D). The correlation coefficient is a complicated and sensitive statistic, see for example http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
Can you provide any context about the type of data this is and why you only have 2 instances for each class (I'm just curious)?
Can anyone help me with regards to my original problem because the problem still persisst?
I can elaborate on the work I did in my MS thesis if needed, also.
Wow, thank you for so detailed answer. Very interesting. Certainly worth to be accepted.
I do not quite follow your question, do you just need the pair of observations with the largest difference?
A coin toss is too simplistic though, since I can't identify a 'surprise' win without taking there having been a higher probability of one player winning in the first place. The normal distribution of strengths was my professors suggestion, as was the use of logistic regression, but I can look into alternatives. He specifically said not to try and make a tennis match predictor, which is what I thought was needed initially. Using just what I have - a strength number for each player and without doing any training how would I find omega and c? What do omega and c represent?
If you just need a quick definition: The top google search on *clustering purity* \\*\\* links [here](http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html) which gives a mathematical definition. (\\*\\* for me, at least -- your individual results may differ)
This last question makes no sense. Perhaps you mean "the response is different only because of group"? but that is answered, the answer is "no, age also matters". Model m2 controls for age.
Do you have any fit indices? Do you have chi-square, for example?
It looks to me that something like a random intercept to a subgroup would be achieved by having a random coefficient ("slope") on the group indicator (dummy).
(+1) The second integral is (by inspection) an incomplete Beta function: knowing this makes further analysis possible, makes many other formulas immediately accessible, and also leads to more efficient computation (when $k$ is medium to large).
*Some* kinds of comparison may be possible, but it depends on what you're trying to do. It's difficult to compare apples with oranges, but some things can perhaps be compared. To be more meaningful than a fairly facile "this number is bigger than that one", it needs to be approached with some care.
@Conjugate Prior: See the comment above.
http://cran.r-project.org/web/packages/CDVine/vignettes/CDVine-package.pdf
Maybe I can help by specifying some alternatives I had to consider when parsing your answer: (i) are you assuming a *known* or unknown population variance? (ii) Ditto for the population mean? (iii) Are you using a z-test, t-test, or something else? (iv) Are you making approximations (as suggested by "approximately") or precise calculations (as indicated by numbers like "16.97")? (v) When you contemplate taking more samples, in what sense can we say the SE of the mean will be halved, given the new sample results change? (vi) You seem to imply a 95% CI won't have 95% coverage.  What gives?
@Max I've updated with a command.
How are you determining the value of the explanatory variable for each group? Are you averaging them?
Not sure this belongs on CV, but it depends on what you are taking the derivative with respect to. If you think to distance-velocity-acceleration functions, the last is the second derivative wrt *time*, and the units are distance's units per time per time, or distance/time$^{2}$.
I think that's always the challenge with any data related work/profession. I'd probably recommend looking at some of your data, if there are variables with descriptions about the data or insurance event, read a couple dozen--get a feel for the data. Remember, all we're ever trying to do is model the underlying process of the data and to really do a great job of that you have to know the data.
This doesn't really belong to CV.SE either, I think.
One thing that I think you've hinted at, but that might be good to flesh out further, is that much of the "delineation" between Bayesian and frequentist statistics lies in motivation, interpretation and philosophy. There are many instances in which either (a) the same exact formal approach can be given a Bayesian or frequentist argument (e.g., ridge regression) and/or (b) so-called Bayesian or frequentist methods arrive at the same solution, though the final interpretations differ (e.g., certain confidence/credible interval constructions).
Thank you. I have asked two Statistics professors and will keep working.
`summary(clm(as.factor(y)~x, link="probit", data=data))`. See the examples for `clm` in the [reference manual](http://cran.r-project.org/web/packages/ordinal/ordinal.pdf), pp 10-11.
Thanks @user777. Unfortunately I am unable to use multi-variate priors at the moment, so I left a follow-up question [here](http://stats.stackexchange.com/questions/94856/univariate-priors-for-the-parameters-of-a-beta-distribution)
Sorry, but I disagree. This is a *screening question* for a job offer that means that the knowledge base of the candidate is being evaluated. If he does not know, it is dishonest to pretend that he does by just foisting off the question to other people, with *zero effort* on his part to attempt a solution. The whole point of the screening questions is to weed out people who don't know. Lots of professors don't know about this site either; that does not mean it is not unethical and a violation of student conduct if students post their assignment questions on SE without disclosing the context.
agree with that easy to read not as confusing as Bishop for beginners.
@pul His inequalities are in the wrong direction: what he claims are $p(n,K,x)$ are really $1 - p(n,K,x-1)$.  Sorry about the bounty problem: I knew how to answer this one when it first appeared but needed to check the results first and had no time to do anything about it until the weekend.
Did you notice that the two models involve *different* dependent variables? It therefore makes little sense to say one is a "better fit." Also, it's clear there is no $R^2$ equal to 39% (it couldn't possibly be consistent with an adjusted $R^2$ of just 0.014): the second line of output is likely corrupted.
A little theoretical knowledge might advance your situation more than any panel of experts.  For starters, one would expect the cube root of weight to have a more linear relationship with height, shoe size, and jacket size, which is important information.  Standard growth charts could be used to linearize the relationships between age and the other variables.  Once approximate linear relationships are established, you can apply the suggestions provided by @Dmitrij or @Mike Lawrence, probably to good effect.