tttjjj/elt_bench huggingface dataset:
Also, I am not sure about all of the quantities you mentioned above, but I do believe that the posterior, under the Jeffrey's prior, will also be invariant to transformations.  See this paper as a reference: http://users.stat.umn.edu/~sudde001/personal_page/jeffreys.pdf
I agree - the idea that there is one "optimal" bin width is a huge simplifying assumption.
This question was tagged with "pie-chart" but it seems to involve some kind of intermediate computation, and your own response didn't address the data visualization issue. Could you update it accordingly?
It sounds like it is about the difference between correlation and simple linear regression.
In SEM, these are called direct and indirect effects. Whether PROC CALIS can estimate them with the standard errors and such, I don't know, but most other SEM packages do this.
@whuber I will change that, thanks.
You must distinguish between two things -- model selection (which type/parameters of the model is best) and model evaluation (what accuracy the model should have on new data). If the first part is present, you must either remove it (with regularization or similar idea) or include it in model assessment CV or you'll get overoptimistic results.
Hi @Bunya, welcome to CV. Could you give a little bit more detail in your question? An example of your dataset could help.
(+1) [Josef Dick](http://quasirandomideas.wordpress.com/) also has some interesting fairly recent related results.
Yes, this seems sensible
Hi all. I can contribute to this based on my recent research. Apparently, a quick way to check the imputted values for Site A over the period 1995 to 2002 is to compare density plots of the imputted data to the observed data. I will continue to look for other checks.
For the wrapper model of feature selection, both training and validation datasets should be supported. In the wrapper model, we are training and testing a classifier given a candidate feature subset. Thus, giving only a training set to this model is not sufficient.
The question is: if the same data used for feature selection is used for comparison but using CV, are we still biased and by which degree?
This website is for questions of mathematical research. I don't see a research angle in your question.
Also: http://stats.stackexchange.com/a/26846/24000
Frank, I'm sure I need to learn more about quantile regression. Before I dive in, I'd like to understand if this methodology can offer some probabilistic component for the choice of the quantile, based on the predictors and the fitted model. For each given set/range of predictor values there must be a likelihood that the actual outcome will fall in a certain quantile region.
If you have a multicollinearity problem then using z-scores will not fix it.  Also, in a regression context it is not necessary for the *independent* variables (of which this seems to be one) to be approximately normal. You would transform it principally to cure *nonlinear relationships* with the dependent variable. I realize these observations don't answer your question, but maybe they might make it go away :-).
P.S. And by the way. If your matrix is similarities, then, well, it's even better. You just treat it as that SSCP matrix I was talking about and do PCA with it. Still, the problem of possible negative eigenvalues remains.
"Distance from the mean" is not intuitively "leverage". Why is that called "*leverage*"?
if you are willing to look at `R`, to my understanding, the `depmixS4` package may be able to handle this.  See here for the paper: http://www.jstatsoft.org/v36/i07/
But you say that $2\\log\\Lambda\\sim \\chi^2_d$. Why don't you use this relationship?
I've found a program online that does a "one sample t-test between percentages" but there is very little documentation with program. have you heard of such a test? it seems to fit the bill here.
Maybe is the lack of sleep, but, isn't that like circular reasoning?
+1. Good catch re: "The expectation is the average value or mean of a random variable not a probability distribution". I didn't notice this subtle misuse of terminology.
Welcome to the site, @Megan. Can you say more about what exactly you need help with? Is there a specific thing, or do you need a general familiarity with the concepts surrounding logistic regression? If the latter, it may help you to read my answers here: [interpretation of simple predictions to odds ratios in logistic regression](http://stats.stackexchange.com/questions/34636//34638#34638), & here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909).
I wonder if you are thinking of MANOVA. If you ran two sepa4ate PCAs, you would only show that the correlation al structure was similar.
Thank you for the answer! It's very helpful!
Thank you, this is so helpful!
Any help out there please? still stuck!
"...so I have been attempting to use qvalue R package to relax the FDR and therefore gain a greater number of significant p-values." *Danger, Will Robinson!* What is the point of gaining a greater number of significant $p$-values here?
@whuber: Yes you are completely right! ;)
Here is the documentation for SPSS Exact tests module. See page 151. ftp://public.dhe.ibm.com/software/analytics/spss/documentation/statistics/20.0/en/client/Manuals/IBM_SPSS_Exact_Tests.pdf
(0) just opinion here (1) no you don't have to do this (2) I've never seen this. Usually if there a multiple models they are presented for research domain specific reasons (3) depending on your field of interest there may be standard ways of presenting models, worth fitting in with the crowed. (4) if doing backward selection (and EPV<50) you might want consider some for of boostrapping: either to validate modeling procedure or try and show stability of variable selection
There is also spikeSlabGAM that has many interesting options and properties: http://cran.r-project.org/web/packages/spikeSlabGAM/index.html
If you are investigating differences between groups, correlation is probably not what you want. Perhaps t-test, perhaps regression.
For the unbounded distributions, the maximum does not converge: it diverges with $n$.  As with the CLT, an appropriate normalization is required.  (This is why it's essential to include location and scale parameters in these families.)  Gnedenko's classic paper on the subject begins (if I recall correctly) by asking whether a series of affine coefficients $a_n, b_n$ can be found such that $a M_n + b_n$ converges.  After establishing this, he then obtains the possible forms of limiting distribution.
Since central moments and regular moments are related via linear transformations, there is no theoretical advantage of one form over the other. In some cases, there might be a small computational advantage in beginning with one form versus the other.
+1 for actually answering the question, and an additional (but virtual) +1 for challenging the quotation, which is provocative but problematic.  I see you are a recent participant here but have already contributed many answers: many thanks and welcome (a bit belatedly) to our site!
Could you supply a reference for this definition of order of a kernel?
@ttnphns Thanks for your comment. It sounds like an answer to me, why don't you post it as an answer...
@whuber I have added some discussion and details. I don't think this is the best approach, but I think some thing can be done.
That was a  nice explanation. Thanks for the thoughtful response.
All answers were both helpful and useful, and would all deserve to be accepted. This one, however, does a very good job at answering the question: with Python, you have to put together lots of pieces to do what you want. These pointers will no doubt be very useful for anyone wanting to do statistics/modeling/etc. with Python. Thanks to everyone!
This was a very helpful answer! The concept of the prior and the data creating something of a weighted average was the idea that was bouncing around in my head, but your examples helped solidify that concept. Thanks!
Got it, what I meant to explain was that - in parametric regression, we specify the functional form in terms of the # of parameters, which parameter. The simple straight line was an example. Did I get it right?
Hey sorry for the additional comment, but the import vector machine seems like a really interesting idea. Out of curiosity, are there any implementations of it?
+1, (of no real value, but I notice you are using the in-line code formatting, you don't like the block code formatting?)
Yes, it's based on information gain. See http://en.wikipedia.org/wiki/C4.5_algorithm.
I don't think I have semiparametric examples.  I am not an R programmer (yet).  Robert LaBudde did all teh R programming in our book.  He gives an example of a parametric bootstrap coded in R for a stationary autoregressive model (pp 120-122).  On page 10 of the book he shows all the bootstrap functions in R that show up using the query">help.search('bootstrap')  Efron and Tibshirani have documented in their book the bootstrap package in R that they call "bootstrap" another package due to Davison and Hinkley is their package "boot" which you can find out more about in their bootstrap book.
Given $z$, you are asking for a *non-parametric tolerance limit.* I do not believe this concept has yet been discussed on our site. (+1) Tolerance limits (or, more generally, tolerance intervals) come in several flavors. There is uncertainty associated with $M$: due to the chance nature of the sample it is possible $M$ will be too low or too high. So you can ask for an $M$ that has a small (but prespecified) chance of being too low (or too high). You could also require that the expectation of $M$ be correct. Do you know which flavor you are looking for?
(+1) this is the only one of the 3 answers (so far) that seems to be attacking the real problem. This isn't about $n$ vs. $n-1$, it's about sample variance (a function of the data set) vs. the value of $E[(X-\\mu)^2]$ - the variance of a theoretical distribution - why should one expect there to be agreement?
Logit and probit models are usually so close that it will likely do no good to apply both. Use just one of them along with appropriate goodness-of-fit tests to detect and diagnose possible mis-specifications. (The coefficients in logit models usually have simpler interpretations.)
First off, I assume the first two distributions you write are independent over $i$? For part (a), Bayes theorem applies to multiple variables - let $x$ be a vector and $y$ be a vector, then $p(y|x) = p(x,y)/p(x)$. Note that $p(y|x)$, $p(x,y)$ and $p(x)$ are all joint probability distributions over the relevant variables. For part(b), your expression of the likelihood is incorrect - your likelihood statement is $y_i| \\text{some vars} \\sim \\text{Poisson}(\\theta_i x_i)$ - i.e. it is a distribution for $y_i$, and note that $\\theta_i \\times x_i$ is simply the Poisson parameter.
Not exactly what I looked for, but interesting nonetheless!
Why don't you expand on what is N.I.I.D and how you would apply it to answer the OP's question?
@Nick : Thank you for the suggestion. Yes, I did look into the function. I will correspond with the authors for the solution. I hope that I will get the answers from them.
This is also mentioned in the betareg vignette as the method of choice, [see first paragraph in section 2, page 3](http://cran.r-project.org/web/packages/betareg/vignettes/betareg.pdf): "$(y * (n − 1) + 0.5)/n$ where $n$ is the sample size"
It's worth  emphasizing something @Nick said: "As all your predictors are also categorical, you will be estimating a lot of parameters". If you're treating e.g. Understandability as a continuous variable, it's one parameter for a linear effect, two for a quadratic, &c.; if you're treating it as categorical it's four parameters (no. categories excluding a reference level). Interactions between IVs add more parameters. Any rules of thumb should be talking about the no. parameters, not the no. IVs. And it's sensible to consider the size of effect you want to be able to estimate.
After some looking, I found [this](http://ssc.utexas.edu/software/faqs/spss#SPSS_22) web site, which appears to explain how to create contrasts in the way you describe in your post.
@AbrahamDFlaxman: I don't think it's to mathy, I think it's great. Memorable, short, and the graph is not just a straight line or anything. The data's well-known and not tied to any particular program, and can be arbitrarily extended by anyone wanting more points. I vote for it!
As stated in the question, the $10\\%$ refers to removing 10 percent of the edges from the network. Making $A$ into $A'$ does not take time and energy, actually $A'$ is just a simplification of $A$.
Yes, what should I do to test for serial correlation? What kind of display would be appropriate?
I would suggest visual inspection of the QQ plot (for each year) to determine appropriateness of the Poisson assumption, and then report $\\lambda$ values.
For us to be able to say what these mean, in any practical sense, you need to tell us a lot more. What are you modeling? What is your DV and IVs? Your last comment makes no sense with your origianl post, which is about a normal distribution and says nothing about time.
(+1) It might help to emphasize that in the case of zero correlation, *every* vector is an eigenvector, so the ones provided at the end of your answer are but one choice, albeit a natural one.
Good catch, @rolando.  If this question were about how to assess interaction in logistic regression it should stand on its own, but it's not: it's really a general question about interactions in regression.  As a check, I notice the replies and comments here would be appropriate in the duplicate question as well.  I think it's best to merge the two threads.
For PCA you would do exactly the same thing for the mixed model. You would just treat each project as having a `Participant` effect. Something like: `lmer(PC1_scores ~ Var1 + Var2 + (1|Participant)`. (If I understand what you mean by your PCA problem.)
Hi - sounds like you are interested in an 'unsupervised' classification procedure - i.e. you don't have a test set or 'training set'?  Some more detail on your problem might be useful.
@MichaelMayer, thanks I will update the code and question accordingly to get answers to 2) and 3)
Thankyou! You've cleared up some of my issues. Turns out $$ \\theta_1 \\text{ > } \\theta_0$$ so that clears up the issue with the indicator function.

So the new test, $$T_{NP}(x) = \\exp(n(\\theta_1 - \\theta_0))*I(min(x) > \\theta_1)$$. Is that equivalent to theta hat?
this is a great response thanks for the help. I may post another question in the future, because this dataset is artificial.
Consult the King. He will not disappoint.  King, G., Tomz, M., & Wittenberg., J. (2000). Making the Most of Statistical Analyses: Improving Interpretation and Presentation. Am. J. Pol. Sci, 44(2), 347-361.
+1 for easy answer to miss formulated question. You should add that this only work if the number of predictor is smaller than, say, 50.
Read starts only once per observation, so $P_{n=1}(X_i=1|X_j=1, j\\neq i)=0$, right?
Could you please provide more details?
It's actually XLStat - I've grown a liking to them...and I can't get away from Excel - ha.  The type of regression I'm attempting to do is something in their software called "correlated component regressional analysis".  More info about it is here: http://www.xlstat.com/en/products-solutions/ccr.html.  I'm attempting to eventually regress (after  possibly a few decision trees) with a binary logistic regression  (it's a part of what CCR can do).

I'll look up multiple imputation, which I'm assuming is forming a model to fill out the data that is missing?  Regardless, I'll look it up. Thanks!
Also extrapolating from iii, it appears that the value .1 falls within our 95% confidence interval for the true mean of b0. So how do we fail to reject the null? This isn't a test of whether a change is significant, in which case I would understand failure to reject the null; it's a test of whether a value is probable.
And of course thanks for the correction. I've edited the answer.
I gives me strange result. I don't know, how to control the confidence area. My result with straight use of your code on my data looks like: [wrong result](https://github.com/matejuh/doschecker_wiki_images/raw/master/consultation/stats/err.png) What I actually needs is when I have some data with kernel density estimation like: [kernel](https://github.com/matejuh/doschecker_wiki_images/raw/master/consultation/stats/kernel.png) And if I have point x[100000,3000] I would like to decide if it lies in azure confidence level or bigger.
Placidia, why did you write "gorm Pearson's" ? I't not anglophone and I just googled it and cant make much sense of it..
I program in Scala.
@user1587457 unfortunately I do not have such document. I would say that you might have to test with several different algorithms and select one which has closest classifying capability (AUC,entropy etc).
Thnanks. Output equation was what I was looking for.
First of all, thank you very much for a detailed answer. Secondly, maybe I should immediately provide some more information.
- It is an in vitro assay, the effect of a drug in cell counts.
- Maybe I didn't explain myself correctly but my intention is only to see if any of the drugs as an effect. Either one or both but they are completely independent. They are not different doses of the same and I'm not interested in combining the treatment groups.
You also need to post this on a programming site like stackoverflow.  Nevertheless: `new.x <- x * ifelse((x %% 2), 3.15, 2.15)` should do it.
@Wayne, B is the number of repetitions. It seems to me that the extreme value is matter. Because of the extreme value the mean in the sample can be higher then 1 and it won't be included in the conferencing interval. But I really have no intuition for it.
Each model asks a different question, so it's not surprising they give different answers. If there was substantial collinearity in the initial model, that model should not be used as a comparison - one common effect of collinearity is reversal of the signs on coefficients.
Regarding naive methods; I have tried this already and the results were not particularly useful. I have now put this approach on hold in favor of trying a Neural Network approach. It appears that for what I want to do that may be a good approach (assuming I can work out how to use them properly).
@bayesianOrFrequentist Take a look at the so-called *Bayesian central limit theorem*.
which p-value do you want? A p-value always refers to a statistical test and a statistical tests always refers to a null hypothesis. So which null-hypothesis do you want to test? "the fit is significant" is not a null hypothesis.
Forgot to mention - for the sake of a priori power analysis, both groups would have the same variance and N; it's just their means that are different, and thus they show different effect sizes for a manipulation.
@AndyW At the time I posted this question I was already familiar with the paper you cite, which BTW advocates using maximum likelihood methods, and has less discussion of the CDF.  But the question is *specifically about estimating the PDF*, not the CDF.
How many rows of data are we talking about here?
Thanks for the answer. I'm not very familiar with R. Why did you set the censored data to NA, instead of 0 (which is what I observe)? Also, re: your comment on using E-M: my understanding was that E-M cannot be used when data are NMAR. Is this correct?
I agree with your second comment. As to the first comment - I'm not trying to compute p exactly. If I was trying to compute it exactly I wouldn't be using monte carlo. However, with $n$ large enough I can (hopefully) make the error in my approximation to p small, which is what I'm really asking about - will this method compute an approximation to the true p value or not?
@whuber Well, that's why I asked the Karen to post her crosstabs, but she hasn't done so.
@DavidMarx I linked it.
Can you give us a little bit more detail on the learning algorithm and architecture that you (or the R package) used? How many layers does the NN have?
Many thanks for that, it gives some extra clarification.
Ah, then @Cardinal is (as usual) correct: the marginals will no longer be uniform and you're not requiring that they be so.  In fact, they can be far from uniform.  (But for medium to large $n$ they ought to be pretty close, because the constraints are so mild.)  What you're asking is this: the constraints along with the bounds of the uniform distributions describe a polytope of $n^2-3$ dimensions; you would like to sample this polytope uniformly.
@allinonemove Of course! Otherwise it would not run.
Doornik-Hansen test is quite well based. e.g. http://cran.r-project.org/web/packages/normwhn.test/normwhn.test.pdf (beware the typo Doornick)
@sarah Please register your account so you could reclaim your question. Just visit this url: http://stats.stackexchange.com/users/login
Relevant reference: [A. O'Hagen, 1990. Outliers and credence for location parameter esimation. JASA 85(409), 172-176](http://www.jstor.org/discover/10.2307/2289540?uid=3739448&uid=2&uid=3737720&uid=4&sid=21101115808171). Short, short version: it's not the absolute value of the prior density *per se* that matters but rather how flat the prior density is in the region of high likelihood, and vice versa.
Can I ask how this handles the `f_color` variable? Is factor level 1 to 4 considered a larger step that 1 to 2, or are these all equally weighted, non-directional, and categorical? (I want to apply it to an analysis with all unordered predictors.)
This question seems specialized to economics.  To achieve wider understanding of what you are asking, please tell your audience what $Q$, $P$, $I$, and $P_m$ mean and how they are supposed to be related to these elasticities.
It's trying to predict tickets sold.
John: if I were you, I would learn basic probability before taking Machine Learning / Gaussian Processes classes. Take a look at this book: http://www.math.uiuc.edu/~r-ash/BPT.html
@user34790  It's not.  It's equal to $2\\lambda\\beta$.  But the 2 cancels with similar 2s on the other terms.  Of course, the $I$ factor is like a factor of 1 in "regular" algebra, you can multiply it anywhere you like without changing anything.
have you ever heard of the geometric distribution?
I'm under the impression that fuzzy numbers assume a distribution (perhaps gaussian) for the fuzziness, so I'm not sure what you gain by adding a second layer of abstraction (the fuzzy numbers). I may be wrong though.
It's unlikely, but the test data is just a finite sample too. Entirely possible to get a good score. Other more malign reasons might include things like the training data being big enough to include more weird observations. While the test data might be smaller and happen not to have any members of the awkward squad in it.
This is an entirely different question, but if there was no good reason (*a priori*) for the third party to use a one tailed test, they would have to double the one-tailed p-value. If there was a good reason to, they could halve the two-tailed one. It would probably make more sense to look at the effect sizes - e.g. look at the difference in means, and the standard error of the difference in means and compare the two sets of results in that way.
I haven't really looked into this, but one imagines that linkedin and specifically looking at the membership totals for UX UI groups on Linkedin would be helpful
This question doesn't make complete sense to me. You calculate confidence intervals for *sample* means. Yes, if the variance is unknown, you should use the t-distribution, rather than the normal. As for 1 vs 2 tailed CI's, you could use either, depending on what is more relevant for your substantive question.