tttjjj/elt_bench huggingface dataset:
@PeterFlom This happens with any variable, but only in the univariable setting. In the multivariable setting everything works as expected.
Picking "5" is arbitrary. Why not "4" or "6"?
Ah I see, that was not clear. On the other hand: perhaps your data simply do not support the analysis you want to perform.
That, plus a trivial Jacobian.
I think you put a $\\cup$ in place of a $\\cap$ when writing the subset/superset bit.
+1 Fair enough.
You could learn a network with 60 samples, but the quality may not be that good. You could use the PC algorithm by Spirtes et al. (or something more advanced like the MMHC algorithm by Tsamardinos et al.) with the Pearson partial correlation test. Consider setting the maximum conditioning size to 1 or 2, as you do not have many samples.
The double-Poisson method would only work if the scores were independent, which they probably aren't.
Just realized I didn't even convert my simulated group data to `0`s and `1`s; `glm.nb` handled `A`s and `B`s just fine.
I also do not have enough reputation to comment, but I have an interest in this question for nonparametric models. So for a particular variable $X_i$, we do not have a $\\beta_i$, but rather a series estimate of some function $f_i(X_i)$. For example, the generalize additive model takes the form: $$
Y = f(X_1) + f(X_2) + f(X_3) + \\epsilon
$$ Would the proposed tool be able to control for the dependence structure in a nonparametric model when the $X_i$'s are known be to correlated?
Welcome to the site, @RichardBlissett, +1 for a good question. OLS regression is a special case of the generalized linear model, where the link function is the identity function & the response distribution is normal (see my answer here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909), for more info). There are 'pseudo-R2's for GLiMs, but they're controversial (see here: [which-pseudo-r2-to-report-for-logistic-regression](http://stats.stackexchange.com/questions/3559/), for more info).
@momeara under current global financial crisis situation, the cost is not much big, maybe my bonus, maybe my job :p OK, seriously, the consideration is not the effort of improving the model, which is the next step, but, a criterion to say whether the model need improvement. i'm submitting a model, not only the numbers, but also a process of model setup, calibration, and validation, where the criterion kicks in. I need make the criterion reasonable, to cover my ass.
Dear confused: the equation above is wrong. Where is the denominator (the marginal likelihood of $x$)? Divide by $\\int f(x\\mid c)\\,\\pi(c)\\,dc$ and you will see that the $\\prod_{i=1}^n g(x_i)$ will cancel out. The "posterior" in the book is wrong for other reasons. Please, check my answer.
How are the two tables related?  Are they two different estimates of the same thing using two different samples?  Two different estimates of the same thing using the same sample (that is, two different methods of estimation applied to the same data)?
I`m setting up a dynamic scenario analysis framework to evaluate the performance of Value at Risk and Expected Shortfall estimation methods. For better realism I want to create several quiet and several stress scenarios where the conditional variance and mean are extracted from quiet and stressful periods in real-world time series. So I can make sure that typical autocorrelation patterns are part of my framework, but also regime jumps as well as slower regime shifts. Only therefore I need the vola.
(i) Maybe $\\lambda$ is given a default value of $1$ as you claimed in a previous comment (?). 
(ii) If you are also given the mean (or the median, or sthg similar), then you can retrieve $\\lambda$.
(iii) If neither (i) nor (ii) holds, then I do not think it is possible to get $\\lambda$.
@probability That is one way to look at it.  Another is to dodge the hypothesis altogether and address the question actually being asked; to wit, *what is the size of the treatment effect*?  From this point of view termination can occur once the estimate is known with sufficient accuracy to support decision making.  For example, we might want to have high confidence that the gain in health from prescribing the treatment is likely to exceed the costs (and side effects) of the treatment.
...  However,  With 0/1 I can't place each persons score along the normal distribution, but over a number of scores a can calculate a probably (or percentage) which i can place on a normal distribution to get a statistical measure of how likely that overall percentage is. - but I need more people to get to this stage.
Again, thank you for your reply. Could you please provide more details about the cleansing process? How were the cleansed values actually computed? (BTW I'm now reading your AutoBox User's Giude, maybe I'll find some info in there?). Also what happened to level shifts? Shouldn't these be somehow incorporated in the model? Or am I missing something? I apologize for my lack of knowledge on this topic...
This question appears to be off-topic because it is about finding an R package.
I have used the option interpolate and extrapolate, and it seems to have done a good job in the sense that the values generated resemble quite well the data, and it generated a set of balanced estimates of the missing values.  I think I will stick to that, and see if I can find some further advise with one of my professors.  Thank you for your answer @Nick!
The problem is that I am looking at a longitudinal study and not at survey data. I think I learned, that I am not able to use simple tests like these if I have longitudinal data. What do you mean by: "the Hypothesis is wrong" ? The hypothesis is the one I have to prove, so if something is wrong then it is the analysis and I have to adapt the analysis to the hypothesis but the hypothesis is given. Sorry I am really confused with my analysis :(
Thanks for your reply.  I've done (1) and it seems pretty reasonable.  As for dealing with the presence of C - I neglected to mention that the number of flips, or number of heads, of C should be fairly constant, which is why I thought including the intercept term might work.
Hi Henry, thank you so much for your reply. I have changed th code, but I keep getting the same problem, he always gives me 1.
There's also "reduced-model" approach where you train a classifier for every pattern of missing values encountered during testing. IE, to make prediction for x where i'th attribute is missing, remove i'th attribute from all instances of training data and train on that. http://jmlr.csail.mit.edu/papers/v8/saar-tsechansky07a.html
I don't get anything like 0.0839. What did you do?
It sounds like an interesting paper (not sure it is freely available, but there's an [R package](http://cran.r-project.org/web/packages/PhViD/)). Now, what *graphical solution* would you suggest?
Maybe you want to have a look at [change-point analysis](http://r-datameister.blogspot.fr/2013/08/changepoint-analysis-of-time-series.html).
Check out @MichaelChernick's [answer](http://stats.stackexchange.com/a/30531/32036) on [What book would you recommend for non-statistician scientists?](http://stats.stackexchange.com/q/421/32036), and [Recommended books on experiment design?](http://stats.stackexchange.com/q/1815/32036)
There must exist an option in k-means to set initial cluster centres. Compare the 2 programs in that regime (give identical initial centres for both programs). Test many times: different data sets, vary _k_, vary the initial centres. The 2 programs should give identical results consistenly. Only then you can be sure there is no bug in both programs.
@Askan, you're welcome.
Isn't one answer apparent at http://stats.stackexchange.com/questions/30565/how-to-obtain-covariance-matrix-for-constrained-regression-fit?
"Either it works or doesn't" appears to skirt some key issues. The concerns with any such algorithm are (1) Does it ever terminate?  Are there conditions under which it will not terminate? (2) If it does terminate, what can we expect of its solution: will it be any good? If it is not guaranteed to be optimal, can we guarantee that it will be within some factor of optimal? These concerns come to the fore when negative "distances" appear in any distance-based algorithm. Another issue is that the `d` of this question isn't a distance at all (but it can be made into a Hamming distance).
The thing which is comparable is the $2\\times 2$ variance/covariance matrix.  You could take the mean of the square of the Euclidean distances between values and their mean, but that is unlikely to be as informative in most circumstances.
I took a quick glance at the GSL documentation and based on what it offers I would presume it cannot do anything like what you ask. It may be worth remarking that what you are attempting is unusual and might be inferior to other procedures. Perhaps you might like to consider modifying your post: describe your data and what you would like to learn from them, then let readers suggest appropriate procedures.
Purely my opinion, but this might be better placed as a single line comment, especially in view of @whuber's remark.
If you don't know how many observations are 'lost' you move from censoring to truncation. But "estimates of higher event numbers degrade" sounds different from both censoring and truncation.
@KerimAtasoy, a big part of this site is to constructively criticize questions/answers when applicable, since we want others to be able to read the question/answer later and learn from it - this is why we've asked you for clarification and why I pointed out the standard use of notation to Michael. Michael and I have these kinds of back-and-forths on here a lot so I'm pretty sure I've not made him feel uncomfortable or guilty. Discussing (or what may seem like 'arguing seriously' about) statistics is something the regulars here enjoy, I hope :)
I have updated what x is. I just don't understand how density of x is given by the increments where xvector=(x1,x2...xn)'. I have also updated what x is. Actually, I got the above derivation from a book called Gaussian Markov Random Field Theory and Applications by Rue and Held. As you said for the above derivation to justify xvector should be equal to (deltax{1},...deltax{n-1}) but I don't think it is the case since xvector=(x1,...xn)
I'm not familiar with `gam` models, but have you examined the different attributes of that object? You can look at the names of the objects with `names(b)`. I'm guessing whatever details you are after will be retained within that object somewhere.
Sorry for delay. You might find useful the paper "Importance of data structure in comparing two dimension reduction methods for classification of microarray gene expression data" from Caroline Truntzer (BMC Bioinformatics). Actually, it is not about factor model, but PCA model.
This is not a statistics question. This would be more appropriate on Stack Overflow. Best of luck to you!
I'm not sure if this would be truly helpful, since this solution is a bit like using a Howitzer to kill a fly, but I know that you could program this in STAN using only examples in the manual. The hangup here is that programming it *efficiently* requires a bit of knowledge of how STAN does its work.
I guess the author is interested in empirical tricks, though more context on the homework task would be great, if for instance I'm mistaking :) Anyway these are nice additions for the 3rd part from my list (+1)
Why doesn't `indexVector` appear in the body of the `meanDiff` function?
One big issue when using income in models is how to treat the unemployed.
Weka can be pretty slow. Try ELKI, it has very fast implementations (in particular when using indexes) and a much wider choice of algorithms.
That's great, and works for me too with my data.  Must be a bug somewhere in my preprocessing script.
My writing this clearly overlapped with @Andy W's writing....
Can you elaborate on your question?  And how do you get an estimate of 12.5 or is that entirely made up?
You need to expand on your chemical-protein interaction example, perhaps giving some example data. Nodes can have many incident edges and you can use multi-graphs or multiple graphs to establish multiple kinds of relations, so I don't see your "only one variable representing the similarity between the samples" distinction. But maybe it's just me. (You'd also probably use different representations for different goals as well.)
Good references, but I don't understand your last comment about a parametric distribution. Splines are usually used for modelling the conditional mean of Y|X and are not directly to do with the distribution of either Y or X. I know you can fit spline-based density estimates, but I doubt that is what Henry was asking about.
Why not just calculate the Mahalanobis distance from $X_{test}$ to each $X_i$, using the sample mean and covariance matrix for the appropriate subset (+1, -1) of the $X_i$?
I can make the graph nicer by plotting the density in R but it was just to give a general idea.

It makes sense to transform this data into something truly discreet. I'm just worried I'll compromise my analysis results this way.
@Manoel: Can you clarify?  I know PSM and nothing he says rings a bell about PSM for me.  It sounds to me like his CRM software has a poor understanding of statistics and probability.
@Glen_b Sorry my bad, jargon can elude me at times. However, the variance is a measure of spread (one of the many).
I see it is clear: "exposure" is another different disease condition.
mpiktas, thanks this is an excellent answer.
Yes, it is a random variable. What is the distribution? I know that the variance of the Normal distribution follows a chi-square distribution (http://en.wikipedia.org/wiki/Variance#Distribution_of_the_sample_variance), but does this apply regardless of the underlying distribution?
@whuber I see the problem now. I think my proposal amounts to defining a markov chain on the state of the grid. Define the state of the grid as $\\{n_{ij}\\}$. Then we have a markov chain on the grid's state as the cell counts of all the cells at any one time period are dependent on the cell counts of all the cells in the previous time period. I agree with you this is an impractical proposal as the state space increases dramatically.
Great answer. It seems that not the difference between the two methods in my post is the cause of the difference but the fact that one of these methods is using continuity correction and the other is not.
@DilipSarwate It is not I that I assume it - the assumption $E[u_i]=0$ is part of the question, so my answer was given with respect to that. Of course it holds for a non zero-mean, and in fact it will be exactly the same, since shifting the distribution does not affect variance/covariance.
It is worth noting that the evidence itself is not affected by multiplicity of testing, even if your response to the evidence might be altered. The evidence in the data is the evidence in the data and it is not affected by any calculations that you may perform in your computer. The typical 'correction' of p-values for multiplicity of testing has to do with preserving false positive error rates, not correcting the relationship between the p-value and the experimental evidence.
Answers appear at http://stats.stackexchange.com/questions/4364. See, for instance, http://stats.stackexchange.com/a/4373 and http://stats.stackexchange.com/a/24591 (the Herschel-Maxwell theorem).
I didn't fully digest the "multiple choice" aspect of the question previously. I would look further into the Van Westendorp methods if that is what you are limited to...
@Proc I read the question as stating that the PDF of $x$, $x\\ge 0$, is proportional to $\\frac{x+\\lambda}{\\beta}\\exp{(-(x+\\lambda)/\\beta)} \\frac{dx}{x+\\lambda}$.
ah okay, that clarifies it already a lot. The study also does not elaborate any further on the tests carried out, so I'm afraid we'll have to keep guessing on that one. :) Thanks!
In econometrics, there is a systematic approach called BLP model studying your problem. A good reference about BLP model is this paper "A Practitioner's Guide to Estimation of Random Coefficients Logit Models of Demand" by Nevo(200).
O yes, I was thinking same, sort of putting into a mixture model. But still, I am wondering how can I choose a proper covariance matrix and set its hyper-parameter. I am playing with a code, and changing parameter without having an idea is embarrassing !
Honestly, I would just do it by hand using the equations you provided just to double check (or at least "eyeball" the table values to make sure they represent what you think they do).  The Weka output is somewhat obscure and gives me a headache every time I look at it.
[This paper](http://faculty.smu.edu/millimet/classes/eco6375/papers/becker%20kennedy.pdf) focusing on a graphical explanation may be of interest. It's difficult to imagine what your plot of the data might look like, because "0", "1", and "2" are merely *codes*--not numbers--to indicate the ordering of the $Y$ values. You could just as well have used $-1000$, $1$, and $1.1$, respectively: but any graph based on these arbitrary values would look quite different. In a precise sense, ordered regression is fitting curves through points on a *trilinear diagram* representing *proportions* of $Y$'s.
By format i mean the expected flow of how you start and end your presentation.
@Rok Great idea for the simulation! I'll look forward for this particular issue. I suppose you already search on the r-sig-mixed mailing and Gelman's book on multilevel regression...
For OLS regression, it is not the case that the 95% CI for $\\hat y$ is $(a_l + b_lx_0,\\ a_u + b_ux_0)$.
The output will look better if you add this to your analysis:
## from the AER book by Zeileis and Kleiber
options(prompt="R> ", digits=4, show.signif.stars=FALSE)
If the series are non-stationary then sample autocorrelations are meaningless, i.e. they are spurious (it is probably possible to construct non-stationary processes for which they are not spurious). So there is no advantage in not holding means and variances fixed.
This question appears to be off-topic because it is about a request for Stata code. Please review advice on software-related questions in the Help Center. This would go better on Stack Overflow (which averages about 1 Stata question per day) and even better on Statalist (which averages many more).
I don't necessarily see anything invalid about doing this, but you have information you are not using when you do it this way. If you have an ordinal outcome, you would do better to do [ordinal logistic regression](http://en.wikipedia.org/wiki/Ordered_logit). The only issue there would be what to do with the `Don't know` responses. Alternatively, you could do [multinomial logistic regression](http://en.wikipedia.org/wiki/Multinomial_logit), which treats them as just 5 unrelated responses; it has no problem with the `Dk`s, but loses the ordered nature of the rest.
This is called a Breusch-Godfrey test for autocorrelation.
The most important question to answer is whether or not when you sampled from each stratum did you do it at random?
Can you clarify this? I don't follow how the first sentence is a question. What is the "tooth takers test"? Can you say a little more about your situation, your data &  your goals? Is this just repeated measures ANOVA, but w/ only 2 groups?
Thank you very much Dennis
@Joshua: My suggestion regarding making the answer CW was due to the fact that the answer appeared via a stream of comments and was generated by a "community" discussion. Since it was resolved in the comments, my thoughts are that it makes the most sense to reformulate it as an answer, marked as CW to indicate this collaboration, and this allows for the answer to be accepted and the question to be marked as resolved. (Otherwise, it will automatically get bumped back up by the software after a certain amount of time.)
A. I don't think it is "naive"; they can reflect valid assumptions. You can even define improper priors if you feel that strongly for something. B. That was only part of my answer, it didn't say it is the only way to go; I gave example for lme4. C. If you need to do multivariate mixed effects it is practically the only available package along with sabreR...
I use the program 'mcxarray' (which is part of MCL). It is threaded and can do job parallelisation (use multiple machines, then combine results). It uses sparse encodings, so N does not figure. I've used mcxarray with M values upwards of 1,000,000. What is your M (and hardware)? I have looked into speeding up computation using the metric properties of alpha (the angle associated with the cosine), but this is really hard - 2-fold reduction of the number of comparisons is the best so far, which is far too little compared to the benefits of simplistic brute-force.
$\\pi$ and $e$ come from Striling's formula
To add a "proof" to some degree to why welch's example produces a "paradox", but in more general terms [here on page 205-206](http://bayes.wustl.edu/etj/articles/confidence.pdf) (page's 31-32 of the pdf file).  This is becuase the distribution used in welch's example is a "location parameter" distribution (i.e. it is invariant under translations).  It also shows that confidence intervals should really be based on the distribution of the sufficient statistic (which may be the whole data set), rather than just a convenient statistic.
Conditional logistic regression and Cox regression are strictly related. http://stayconsistent.wordpress.com/2009/01/25/cox-model-and-conditional-logistic-model-back-and-forth/
Can I ask why do you need a 6-4-4-3-2 NN?
What do you want to classify?
+1 good question
@Alex Chamberlain because he is good :D
Could you please edit your question to include your comment?
Is it possible to use chi-squared goodness of fit on the validation set?  If so, what sort of distribution should I compare my residuals against?
@whuber  if you please, I need the detailed calculation (it's complicated..)
+1, good idea to relate matrix to measure. This seems very appropriate for OP concern about using matrix norm.
Exactly the answer I've been looking for!
OK, that's easy.  For the record, the solution for an arbitrary number of unspecified $c$ is given at http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Setting_confidence_limits_for_the_shape_of_a_distribution_function.
Calibration samples are usually either prepared in a way that ensures a known concentration or the analyte concentration of the calibration samples is measured by a reference method. In any case you have samples where you have the known concentration and the instrument response. Which way you took experimentatlly usually doesn't matter for the further data analysis.
@Azula R. I did not explicitly enter the age variable in my model as 'age minus mean age', but the linear predictor as a whole is centered at the mean values of all predictors (also see [link](http://cran.r-project.org/web/packages/survival/survival.pdf), page 49) when making predictions.
It would be $\\Phi(z)-\\Phi(-z)$ for part 1.
you could add that when the mean and the covariances have to be estimated, then, the mahalanobis distances [have](http://stats.stackexchange.com/a/20549/603) a rescaled beta distribution
@Robert Frank: by asking this question you are effectively helping out anyone else who will in the future have the same doubt as you. :)
ML depends only on one parameter? You say that it is constrained to interval [0,1], yet in the graph we see values from 0 to 50. If you do minimisation, then the minimum should be achieved at point (30,-210), are you sure you did not confuse minimisation with maximisation? From what I see, you either confused minimisation with maximisation, or your optimal solution lives on a boundary. In the latter case gradient based methods are not guaranteed to work.
See [Interpretation of R's lm() output](http://stats.stackexchange.com/q/5135/32036).