tttjjj/elt_bench huggingface dataset:
Plot the dependent variable against each IV using scatter or box plots for continuous and categorical IVs respectively . Look at pairs of IVs using either lattice plots in `lattice` or faceting in `ggplot`. For the model, try the effect plots in the `effects` package.
I find your answer very useful! Thank you sooooo much!
Good to know the DBA buzzword, this may prove helpful in itself! Thank you!
Don't have any better recommendations. Would consider trying a more complex classifier (logistic or RF) to quantify loss in prediction from using a tree. May not be much, may just be a difficult problem. Depends on your situation, but can sometimes turn a model into simple graphic representation. Point systems, nomograms, and above mentioned grids (for 2 variables) have been used to make models easier to use (?interpretable).
@Dail I think you already got an answer for your question as originally formulated; it would be better to post your follow-up question as a [new question](http://stats.stackexchange.com/questions/ask).
I do not understand the example.  When you are comparing two families of models $H_0$ and $H_a$ between them do appear to exhaust every possible model in the family.  If you allow the null and alternative *not* to cover every such model, you complicate the process of evaluating the decision-theoretic risk of the test (both in theory and in practice).
You might want to look at the US Census Bureau's X13 software https://www.census.gov/srd/www/x13as/
Use the fact that $H=H^2$ -write out what that means.
Classification accuracy of a positive only test set is a **meaningless statistic**. For example, a classifier that always returns 'positive' will perform at 100% accuracy.
@Diego Make sure you are clustering only by rows/columns, not by both (default is by both).
I'll add that an issue with approach b is that it doesn't account for statistical significance. Intuitively, a dataset with two points would generate a line-of-best fit that has an r of exactly 1, but we both know such a value is not statistically significant. David Robinson's test statistic for inference on the correlation coefficient is probably the most valid approach.
your definition of the function $f(x)$ is missing an $x$ on the right hand side.
What type of model do you use? A Poisson-GLM? Your question is solved by using the hyperlink dummy and the product of the dummy with the position as regressors (like an interaction).
Papers which use entropy minization are 1. Deniz Erdogmus, Member, IEEE, and Jose C. Principe, An Error-Entropy Minimization Algorithm for
Supervised Training of Nonlinear Adaptive Systems  2. J. Principe, D. Xu, and J. Fisher, “Information theoretic learning,” in
Unsupervised Adaptive Filtering, S. Haykin, Ed. New York: Wiley,
2000, vol. I, pp. 265–319.
As far as I know "Poisson residuals" refers to residuals from a Poisson regression, not a multinomial logistic regression. Do you perhaps mean some other kind of residual? 

(Also, welcome to the site; I removed your signature since the site adds it automatically).
There is probably an element of truth in your reference to WWII. An interesting nugget of information from wikipedia is: "During World War II, he (George E. P. Box) performed for the British Army experiments exposing small animals to poison gas. To analyze the results of his experiments, he taught himself statistics from available texts. After the war, he enrolled at University College London and obtained a bachelor's degree in mathematics and statistics." Great question, by the way.
@Chap, that's cool.  Good luck!
Subhash, you now have no fewer than *five* separate accounts.  Please contact the SE team to merge them: http://stats.stackexchange.com/help/merging-accounts.
If we don't set any `rng`, we have different initial weights and indexes for cross validation. Can we compare different structures of neural network and so final chosen models? ( we are using all `number_of_loops*number_of_kfolds` models as a black box and average between outputs of these best models (with lowest classification error)
I see round 2 is more difficult :) Mathoverflow may give you a faster answer than stat.overflow...
Chapter 3 would be of interest to understanding rates, http://goo.gl/H4lcD , (Rothman, 2002)
It would be helpful to modify this answer to include the *very strong* but unstated assumption that the sample variance follows a chi-squared distribution when the underlying data are *independent* and follow a *normal* distribution. Unlike the theory of the distribution of the sample mean, where in practice its sampling distribution will be *approximately* Normal to reasonable accuracy in many situations, this same asymptotic behavior tends not to happen with the sample variance (until sample sizes become extremely large).
This can be found in any textbook on nonparametric econometrics; see [Li & Racine (2006)](http://press.princeton.edu/titles/8355.html), for example.
@user785099: In general, no. The Dirichlet Process is a different, more general, animal whose definition depends on the usual Dirichlet distribution. They become the same only when the sampling space is finite (which isn't the case of your problem). You should read carefully the appendix of the paper linked above.
Looking into it more, your suggestion would have worked well Greg if I had better prior information. Thanks.
Is the order of the switches relevant? I'm thinking that you might just have 12 different values: the number that made a switch (at any point in time) from cat.1 to 2, 3, or 4, cat. 2 to 1, 3, or 4, and so on. Then you could visualize this with four circles for the different categories, and arrows going back and forth between the circles, and the relative size of the arrows showing the number of switches.
I think it's unfair to include R and S in the same list as those other languages in wikipedia - there is nothing you can't do in R that you can do in python.  Of course there are many things that are better suited to another programming language, but the same is true of Python.
It is relative expression of target gene, which is, as expected, different for different time points, but is One-way ANOVA sufficient test to statistically confirm or not the difference?
**Hint**: What is the distribution under the null of $-\\log X_i$?
Hi Stephan, I have added the data and the plot for my dataset. As you can see that I have only 27 data points, so I am running on fumes here. Could you suggest something here.
To make this a statistical question (instead of a software-specific question which fall under the category of "too localized" question), you probably want to broaden the scope of your question and ask how to compute generalization error for a particular dataset and algorithm/classifier.
Have not. But that was my first instict. Log or square root transformation. But I wanted to see if anyone had any experience on this.
6 parameters aside from the intercept for $n=14$ seems to strain credibility.
This question appears to be off-topic because it is about asking for an R text corpus.
Although it is impossible to determine what the question is (because the problem you want fixed has not been clearly described), your text and comments seem to suggest you are misinterpreting the lower endpoints of the error bars.  I would like to suggest you research our site about the meaning of [confidence intervals](http://stats.stackexchange.com/questions/tagged/confidence-interval?sort=votes&pageSize=50) because that may resolve concerns you might have.
Bayesians may make the same assumptions about the error terms in linear regression but the inference depends on priors on the regression parameters.  Frequentists don't use priors.
The answer should be data dependent. Are you asking how to determine them?
I notice you are using base-10 logarithms in your calculations.  This motivated me to check that your matrix is, up to the limited precision in which you report it, exactly $0.30$ times the one given by `R`. Finally, I see that the [Wikipedia article](http://en.wikipedia.org/wiki/Tf%E2%80%93idf) takes pains to state that "mathematically the base of the log function does not matter and constitutes a constant multiplicative factor towards the overall result," suggesting that only *relative* tf-idf values have any meaning.
Can you give us the context of your problem? What assumptions are you making?
@Bitwise: please see my edit
I am sorry, but this question makes no sense to me. Normally one would group the scores in order to see how the teams are distributed across the scores. Your grouping, as far as I understand you, is purely artificial. What shall this plot demonstrate ?
I think this is the naive approach I'm outlined in my question. Is there an smarter (i.e., more efficient) way to do this? For example, can I uniformly sample any `[an, bn]`, and calculate subsegments, and uniformly sample again within that smaller space? Will two uniform samples like this give me the same distribution as one uniform sample on the entire space of subsegments?
@Roland i thought this question was unique enough asking for a program which does it rather than a programming technique to warrant a separate question.
Is it a strange idea to divide length of imprisonment by age to calculate the proportion of their live they spend in prison? I could than center that proportion and only within the offender group see if this variable influences the interaction between A*B.
Maybe you only need to say that : $|b|\\neq|d|$
"Another thing to ask is what is the range of X values in your data?" This variable ranges from 2 to 6.
Can you clarify your concern regarding the non-normality? Are you worried that the raw response data aren't normal, or that the residuals aren't? (NB only the residuals matter.) Is it that the response is categorical (eg, 0 or 1), or a count? What do you mean by the *parameter* being non-normally distributed?
Hmm I dont think I am whitening the data, that would imply all the eigenvalues are the same. I do think that I am diagonalizing it though. (ie, making the covariance matrix diagonal).
Are you saying, in effect, that you have just *six* data values? (Rural populations in 1980, 1990, and 2001 and urban populations in 1980, 1990, and 2001.) If not, please describe your data more fully.
@rocinante I'll wait and see if somebody has any suggestion about this, otherwise I'll try again on quantitative finance. Ty!
Do you mean transforming data from wide format to long format?
@ScottStafford, I added a copy of the plot above.  This one uses qqplots but the function will also generate histograms or density plots could be programmed.
I don't think this question is about `stan`. If I understand correctly, you essentially have only two options. One is to delete the incomplete information you don't care about (probably wasteful and fraught with hazards), and the other is to model the missing data explicitly and average over its posterior when summarizing quantities of interest. But you'd face this choice using any software tool.
Thank you for the OCR scanning, whuber!
Thank you. I'm afraid that in terms of data, this is it. Also, I should mention that I'm constrained to the multiple linear regression and that my situation is such that I cannot perform any more diagnostics or run different tests, so I have to interpret what you see here. Many thanks as always.
@Makato- I'm afraid your approach is misled. Performing random predictions wouldn't reveal such a bias- random predictions (of 1/3 each) will always have an accuracy of 1/3, just as your random test did. The best that a dumb algorithm would do is the maximum frequency of a branch- for example, if the branches are .5-.3-.2, the best you could do is .5. You need to use that value rather than the value from random predictions.
Also help with laying out tables on this site is much welcomed and appreciated.
I would probably just extract the t- or p- value and calculate that myself. For example,  `summary(mod1)$tTable[,"p-value"]`  The t-value would remain the same and the one-tailed p-value is either 0.5*(two-tailed p-value) or 1-0.5*(two-tailed p-value) depending on the direction.
See http://www.stat.columbia.edu/~gelman/stuff_for_blog/oh_no_I_got_the_wrong_sign.pdf.
Thank you. I may not have explained well - I have posterior samples already, from several imputed datasets, and I want to know if I can simply combine these and then form a multiply imputed credible interval ?
Velcome to our site!  But you should try to write a bit more explicit response, defining terms (such as "propensity score matching method") or at least some references
Also I updated my question a little with new findings.
Where does the a priori notion of what is expected come from? This is often the role of the null hypothesis in hypothesis testing and "detecting fishy data" is equivalent to rejecting the null. In your example, you'd certainly start to think perhaps the die was not fair (i.e. reject the null hypothesis that it is fair)
(+1) I would like to suggest--and emphasize--that the value of this contribution lies in the *principle* it uses: we can almost always reduce probability calculations to *counting.* People tend to reason inaccurately about probabilities, yet very accurately about counts. In fact, people whose job it is to communicate probabilities (such as those who talk about risk in human endeavors) have learned to rephrase probabilities as counts or relative frequencies.
also can I clarify what you mean by "the subclasses' label determine the final Y".  I presume you do not mean literally "determine", but that there is some unexplained or random element of Y?
(1) Why do you need the inverse? (2) There are some very common algorithms for matrix inversion. If the matrix is sure to be symmetric positive definite, you could use Cholesky decomposition (it's relatively easy to invert the triangular factor), but [there are more stable approaches](http://math.stackexchange.com/questions/86568/inverse-of-a-symmetric-positive-semi-definite-matrix) that are suitable even if it's only positive semi-definite, or nearly so. If you have the original variables from which the covariance is computed, the QR decomposition is generally a better place to start.
@BondedDust is right that this isn't really a programming question. As a first step though, table your variables like `with(ihd, table(ihd,gender,drink) )` - that will give you the numbers in each group.
@AdamO dealing with permutation tests under unequal variances is a problem; how do you calculate the permutation distribution under the null without throwing out the unequal variances you started with?
If you hold one of $\\alpha$ or $\\beta$ constant while the other goes to $\\infty$, then it will be centered at one end. In my e.g. there, $\\alpha=\\beta$, so the mean is at 0.5 (and it's symmetric). If they both go to infinity such that $\\alpha/\\beta=k$, then the delta function it approaches won't be at one end, but at a value that's an obvious function of $k$.
Is the $\\sim$ relationship not what you're looking for? If so then that should probably be in your question for clarification.
@davips, the [-1:1] normalization is performed at the input in ELM, but yes you also need to normalize the output because a uni-polar sigmoid function ranges at (0,1). And logistic regression is a good approximation of bayesian prob, you can use softmax for multiclass problems. You can also use one-vs-all strategy to treat multiclass problem as a two class problem.
Thank sir, Because I want to reduce overfitting problem so I want to add some penalty term in Loss function. Do you know other kind of simple penalty term or regularization term to reduce overfitting problem in linear classification that published
(+1) I'm glad you were able to identify the mistake on your own. I had planned to write up an answer to this effect, but have been pressed for time. :-)
It's a card game, Dominion. Players start with a generic deck, and have the choice to purchase cards to make their deck more powerful, and eventually buy cards that give points. There are over 200 cards with different abilities/synergies/antisynergies, however in any one game 10 cards are chosen at random to be available for purchase. Obviously, some cards are better than others and contribute more to winning, but I'd like to have an objective, statistical way of knowing which ones are better.
@Pegah  $b = (X'X)^{-1} X'Y$  is just one way of saying that $b$ solves the equation $(X'X)b = X'Y$.  Symbolically replacing $(X'X)\\to A$, $X'Y \\to B$, and $b \\to x$ shows this is *identical* to $Ax = B$.
If the authors' meaning is not clear to you, I suggest you contact them directly. My impression is that they are using the same convention as I have described, but I am not going to read the whole paper or do further work to check what they did.
Er, "...if the *link* no longer works." - my keyboard is misbehaving and dropping letters.
given $Y_1$ and $Y_2$.
I intentionally did not mention the article exact name and which non-sensical variable is used. If someone feels otherwise, feel free to change.
And one further question; why would you classify what i am trying to do as functional data analysis exactly? Does that not refer to the properties of the way the images are represented (i.e. the color values), instead of what i am actually analysing (the images as a whole)?
This is a **very** broad question about variable selection. It would be helpful if you asked a more specific question. Or, you can see one of the many other questions on this site concerning the topic of model selection - http://stats.stackexchange.com/questions/tagged/model-selection?sort=newest&pagesize=50
If your table is typical why are the data aggregated? This is not my field but it seems likely that the capacity is known precisely for every ship, in which case you could bootstrap from the known distribution. Note that your restriction does bite you: if you don't want to specify a particular parameterised distribution, fair enough, but is there a third alternative beyond resampling the data?
This answer summarizes a talk I prepared in December 2008 and posted on the Web at that time in PowerPoint format.  Its conclusion appears to be substantially similar to Groisman's, even though the justification may be different: "If we mean ‘This awakening is a Head-awakening under setup of wakening’, then her answer should be 1/3, but if we
mean ‘The coin landed Heads under setup of coin tossing’, her answer should be 1/2."  See http://philsci-archive.pitt.edu/3382/1/SB_PhilSci.pdf.
For this and other reasons, people generally work with the log of the likelihood function. Search for log-likelihood and you'll find a crapton of stuff.
For starters, your random variable is not truncated -- it is censored above.
It sounds like you want a "prediction interval".  The answer (with formulas) is available on this site [in several places](http://stats.stackexchange.com/search?q=%22prediction+interval%22).
@Angelorf could you please review these instructions? i am using batch version. so for the 1st batch i find v0-h0-v1-h1.now we find +ve and -ve samples and update gradient. Then for next batch, gibbs chain starts at h1 of first batch inplace of h0 of second batch. Am i right?
I am puzzled by the conjunction of *time series* and *distributional tests.* The latter ignores the time element altogether, so questions about the spacing of measurements would seem irrelevant and it seems that you are in a standard setting of comparing one batch of data to another batch. The issue of missing data is a separate one which can be addressed independently--but if you use time-series methods to impute the missing data, you are already assuming your datasets are not groups of independent results, which greatly complicates the distributional testing.
Appreciate your feedback and the links! Upvoting.
I will have to try that. Sounds interesting. Thanks!
Could you edit your title to be more descriptive? This would provide a better incentive for users to offer you a good answer.
Thomas, Howes, Luk - 2009 - A comparison of CPUs, GPUs, FPGAs, and massively parallel processor arrays for random number generation. http://doi.acm.org/10.1145/1508128.1508139.

Discussion + benchmarks of a set of PNGs executing on CPU, GPU, FPGA and Massively Parallel Processor Arrays.
Though I haven't checked every single step, the reasoning appears sound. That makes the question rather hard to answer with anything but "yes". But I'll add a challenge (hoping your course is not entirely theoretical): can you imagine a way of checking (by using a computer) whether your final answer, 3334, is right, without advanced reasoning or maths?
I always seem to forget about dotcharts, even though everywhere you look Tufte talks about their superior data-ink ratio... I think this suggestion combined with the reordering of the data makes for an informative and easy to digest graph. For completeness sake, to change from bars to points in `ggplot2` simply needs the  `geom = "point"`: `qplot(x = names, y = freq, data = df, geom = "point") + coord_flip()`
Actually, Matlab's PLSREGRESS function provides a full basis somehow!  Is there a listing of the algorithm or what is happening from a linear algebra standpoint?
Sorry, I should have added more details before. I updated my question, PTAL.
What i really want is that my new predictions follow a pattern (levels could be different) closely.. So if the origin is shifted, my predictions should follow historical observations. Any leads would be appreciated.
My institution has an electronic copy of Louviere et al (2000). "Stated Choice Methods: Analysis and Applications". Cambridge University Press. This is the reference that Lee (Life Science Journal) cites for rho-squared in {0.2-0.4} = "VG fit". On page 55 of Louviere (associated with equation 3.32) we see the following quote: "Values of rho-squared between 0.2-0.4 are considered to be indicative of extremely good model fits. Simulations by Domenich and McFadden (1975) equivalence this range to 0.7 to 0.9 for a linear function".
I don't have too much to say on your exact question, but would advise you *not* to use red & green together in the same graph. Red-green colorblindness is common in men, ~10%. [This](http://jfly.iam.u-tokyo.ac.jp/color/index.html) page has a lot of good, relevant info. If you scroll down, it lists sets of colors that are unambiguous for all, w/ coordinates for the colors.
If this is homework or study related you should mark it with self-study tag
I will say similar questions do come up when people do statistics (and some books answer some of them), but most of them aren't actual questions relating to skepticism - mostly they're simple comprehension questions. e.g. "what are degrees of freedom and why do we care" isn't skeptical at all, it's asking for clarity of explanation and motivation. The reasons why advanced users of statistics don't worry about that is because they already understand why they're so valuable.
Can you develop a little on the variants of $k$-means? I had a quick look in *The elements of statistical learning* (Hastie, Tibshirani, Friedman), chapter 14... they support the idea of existence of a "$k$-means algorithm".
he was a mathematician that was very important to statistics. Half of the men on here lived before statistics was an "official" field of study .
@ Henry the Guardian Data Blog is a good place to go for some of these kinds of statistics http://www.guardian.co.uk/news/datablog
I don't get the point of either question, because both are answered in the introductory paragraph: for question (1) set $\\mathbf{a}=(0,\\ldots,0,1,0,\\ldots,0)$) and for question (2) pick any $N$ linear combinations $\\mathbf{a}_1, \\ldots, \\mathbf{a}_N$ you wish to create a set of vectors $\\mathbf{y}_i = \\mathbf{a}_i \\mathbf{x}$ having any desired (valid) covariance matrix. If this isn't perfectly clear, more information can be found in [hundreds of threads about the MVN distribution](http://stats.stackexchange.com/search?q=multivariate+normal+distribution).