tttjjj/elt_bench huggingface dataset:
But see there's the whole chicken and egg thing where I can't estimate my $b_i$'s without having my $P(b)$ and $P(a)$ and I'm not sure how to go about making good starting guesses. Perhaps I could assume the priors to be equal for the first few $b_i$ calculations? I will make sure to try several starting values for the parameters of the gaussians
My understanding is that a weakly-informative prior expresses more about the researcher's attitude towards the prior, rather than any mathematical properties of the prior itself. The canonical example would be Gelman's recommendation of a Cauchy prior with location 0 and scale 5/2 for logistic regression.
Only people with Miles in their username may answer this question. :)
A logit transform does keep it between zero and one, but the logit asymptotically approaches zero or one, meaning any model using the logit transform would shy away from hitting 0 or 1. My data doesn't behave like that - when a queue gets empty, it gets empty.
That was explained [e.g. here](http://stats.stackexchange.com/q/50745/3277), read the answer and the comment. "PCA method in FA" is "Principal Axis method". It can be iterative (usually) or with just 1 iteration (rarely; if you know true commulanilies in advance). Once again let me recommend you to read a good book or chapter on FA instead of asking such sort of questions.
If the deliveries are every 3 months, and the data in the question are annual, then the average sales between deliveries is 95/4. So you want y such that 1-POISSON(y,95/4,TRUE) is smaller than 0.05. Choosing y=32 will give the out-of-stock probability of 4.1%.
@DikranMarsupial: (unfortunately, for most of my data, I cannot use the analytical leave one out as I have nested/hierarachical data structures). Also, I've met some ugly surprises with leave-one-out so I do iterated k-fold or out-of-bootstrap validation. However, that's off-topic for this question.
But the variability of results among one method is dominated by the variability of tests, so I am afraid the spread may be negligibly meaningful.
Can you provide the citation or link to the text you quote?
Hi @StasK, thanks for your answer. I will would study your paper carefully. I have actually removed variables with 0 variance. My question is, wouldn't items with very low variance lead to underidentification as well? If so, how low should be considered low? My second question is that if I were to use polychoric correlations, can this be used as an input to the `sem` package? (I've checked the help file but it only allows input to be covariance matrix or the full dataset) Thanks so much for your time and help.
"Correct" is not the right way to look at it. It's one way (of many) to decide between distributions that are more-or-less consistent with the data and ones that clearly aren't. There is literally *no* way in which to decide what the distribution that generated the data was, if you don't have the entire population.
I assume you are referring to removing variables without using the data at hand.  You cannot use the dataset at hand to do this.  This would be unreliable and distort statistical inference.
The values of $v,w$ and $t$ such that $h=0$ depend on the initial conditions.  Are you considering those to be fixed?
Abbreviation LDA has several different meanings. Please avoid using it without explaining what you mean first (in particular in the title).
@Aly [here](http://stats.stackexchange.com/q/37743/603) or also [here](http://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_pseudoinverse#Singular_value_decomposition_.28SVD.29)
If you have a series that exhibits 1,2,3,4,5,7,9,11,... there has been a change in trend at period 5. The main question in time series is the detection of level shifts e.g. 1,2,3,4,5,8,9,10,..or another example of a level shift 1,1,1,1,2,2,2,2, and or the detection of time trend breaks.
GREAT! How about you edit your answer to make that more explicit, and possibly offer a link or two to the requested "best practices" or guidelines? Or create a how-to right here. :)
(+1) Excellent explanation, with clear articulation of two approaches which cover a wide range of use-cases.
http://www.ebyte.it/library/docs/math04a/PdfChangeOfCoordinates04.html is a helpfull explonation of whut I was doing rong.
I've performed a simulation study with a fairly large number of datasets, and have found that using a large number of hidden layer neurons and weight decay is not a reliable procedure (at least using Bayesian regularisation).  Using regularisation and choosing the hidden layer size seems to be a better approach.
@Nick Cox I didn't know that. For econ specifically, I think the Gini inequality index is what most(?) people think of when you mention the Gini coefficient. The Gini coefficient though is not really all that similar to the Simpson index, though.
Hi, thanks, I think I get the idea of this approach. Do you have some R code or a package that has this implemented? My skills are not sufficient to write my own package from the paper you provided.
I was going to do ggplot2 myself, but thought perhaps lattice would be an easier step. Either one provides a structured way to stack graphs like this. Tweaking a lot of details might get hard, but getting something attractive and readable quickly is easy. (Though you do have to wrap your head around creating the right data frame from which to graph the data, instead of just throwing stuff into the graph.
@siew To get a proper comparison of solver performance, compare your `quadprog` implementation with libraries like LIBSVM or SVM$^{light}$.
There could well be only a single cluster in your data set. You can visualize 4 dimensions still pretty good. Do you **visually see** more than 2 clusters?
you might want to expand your questions to include (1) number of predictors examined (2) method of examining predictors (feature selection) (3) number of observations (4) motivation behind building model
I have access only to abstract. What is the concept of the compositional and hierarchical approach to time series?
I do not know if Mark and Shern had your view in mind but just to re-phrase your point- if the model for the data under the null is 'wrong' then you will reject the null hypothesis for a large enough data.
Yes you can. They are interpreted the same as they would be in other regression models.
I still need to think about the contrast. I will correct it whenever I get time. Sorry about that
Thanks! So if I want to validate a orthogonal-rotated factor-loading with CFA (as I have done), it makes sense to validate that with a CFA model where the factors are uncorrelated. Since it's not a great fit, you are saying I should re-run my EFA using an oblique rotation, and then build a CFA model where the factors may correlate.
I am downvoting, since this is a easily googlable question and it does not conform to guidelines in FAQ.
Good answer.  It might be good to point out that the likelihood has to be gamma-normal if the estimated distribution is normal.
"This should be explained in your favourite (econometrics) textbook." -- these textbooks will also discuss the use of conditional logit (`xtlogit`)
How are the DVs defined - nominal, ordinal, continuous? Also, with only 12 observations, why not post your entire dataset?
@JaakkoSeppälä, how can the algorithm know if there are 17 correct digits? unless it knows the exact correct answer, but then why iterate at all?  These algorithms only look at the degree of improvement in the last step.  Since machine precision for a double precision floating point number is about 16 or 17 digits it is not reasonable to expect meaningful steps/comparisons at 17 digits.  I am not familiar with Sage, but I would guess that the convergence criteria is probably at around 8 significant digits.
First, the notation $E(Y_i = \\pi_i)$ makes me think of the expectation of the indicator that $Y_i$ equals $\\pi_i$. I think what you want is $E(Y_i) = \\pi_i$ and analogously for the variance. Second, if the values of $\\alpha,\\beta$ are being treated as known, I don't see how a confidence interval comes in. Re: your actual question, I don't think testing whether the average predicted probability is the same in both samples gets at your question. Very different coefficient values could produce the same $\\hat \\pi$. The target of your inference should be the coefficients themselves.
Honestly, I don't agree. Scripting is much more powerful.
@whuber: Not everybody who interacts with statistics has the mathematical background to understand his answer.  What I see on Zen's answer is a mathematical proof which is great.  Much better than my poor person's approximation.  However, it is also an answer that I personally wouldn't expect to understand even in my wildest dreams, because the explanation is based on equations rather than a narrative in a language I can understand.
I think this is different enough from the old question to merit being kept open. Of course, since it's been on the board since July, it may not get any answers.
Stating the software you are using would likely help. This is such a simple question it is hard not to suggest to just read the manual, [this meta thread](http://meta.stats.stackexchange.com/questions/793/internet-support-for-statistics-software) gives lists of useful links to various software packages.
I'm assuming that this was written as some sort of a joke.  It should simply be removed as it not only doesn't add anything of value to the site, but actually makes us all dumber for having read it.
I wouldn’t say that you can conclude it with ‘95% certainty’ (Google for ‘correct interpretation of confidence intervals’). Also, this is based on the assumption of *independent* trials with equal success probabilities, which may not be realistic here. Perhaps the last panels installed had a higher risk of being incorrectly installed (the person installing them was getting tired/bored). Or perhaps the first ones were, since the person was less experienced then. Anyway, if the architect was told to test if *all* the panels are correctly installed, he should do his job, not just test a sample!
Question 1:  Is $E[x] = \\theta$?

Question 2:  
a) Look carefully at the expression for $\\hat{\\theta}$.  Does it resemble some other expression you may be familiar with?  If so, can you use what you know about that expression to help you answer the question?
b) Reread whuber's hints.
Yes, And I found that If I count the Previous again. It cann't  be converged
All portfolio management books that I have referred to specifically used the cov of the factors.. I am a bit confused..
@whuber You are correct; you'll need the support of $X$ to be in the ROC of the Taylor series, so in particular, $0<X<2 \\mu$ almost surely. I'll update the post to reflect this.
Dear Michael, many thanks for the quick answer. By p-value adjustment I assume you mean "bonferroni adjustment", right? Many thanks, Chris
Atilla's answer is largely correct (with the qualification I supply as a comment), but I thought I'd briefly note that in the specific case of chess, it has been demonstrated that Deep Thought plays chess very differently than chess experts. Deep Thought explicitly computes many moves in advance. Experts only compute a few moves, but they also employ the computational shortcut of *remembering* previously experienced games and outcomes. This remembering manifests via implicit pattern recognition that overlays an affective experience to the game (certain moves "just feel right").
@Glen_b added, thanks I'm new here.
@Julian, while you're reading, be sure to also check out [Mikosch's critique](http://www.math.ku.dk/~mikosch/Preprint/Copula/s.pdf) of copulas. The article was published with several discussion responses, many of which are quite interesting in their own right.
@guest Thank you for correcting this answer. However, in spirit Dr. Chernick is right--as you hint--because the correlation coefficient is a terrible description of this bivariate relationship.  The high value is due to a small number of low outliers; the remaining data exhibit little correlation.
The $F$ distribution has some nice monotonicity properties associated with it. If you want, you can look at [this question and answer](http://math.stackexchange.com/questions/26632/monotonic-behavior-of-a-function/26649#26649). I'm not quite sure the paper there will be immediately relevant to your problem, though.
Subjects are being taught to communicate using a letter board. Every time they point to a letter, we score a "response."  Built in to the score is whether it was successful or not.  We're also marking every time subjects tap the table or any surface that isn't the letter board.  So then we have a count variable of "irrelevant taps" and a ratio (successful responses / total responses).   To get correlations we correlated TAPS with SUCCESSRATIO for each subject (e.g. subject 1 is [TAP_1 TAP_2 TAP_3 TAP_4] vs [SR_1 SR_2 SR_3 SR_4]).  There several other variables, but this is general form.
All of this seems to imply that the covariance of the sample $cov(X_i,X_j)$ can effect the precision of the regression coefficients in a significant way. Does that mean that it could be better to use a subset of the samples for witch the sum /or sum of the absolute values of the of diagonal $cov(X_i,_X_j)$ is smaller, then for the full sample X?
Didn't know pscal handled classical NB models (besides zero inflated and hurdle).  Thanks.
I'm using SPSS, the first link is very helpful though, thank you!
What you're discussing would normally be called 'deleting' or 'removing' rather than 'dismissing'. The effect depends on how the points are chosen. Choosing at random would lower power but might otherwise do little harm. Choosing in other manners can have far more serious effects.
I have edited my post to answer this
I tried to fix the TeXing - I'm pretty sure this is verbatim what you had before.
The Bank of England releases density forecasts of inflation. You can find more info here: "Evaluating the Bank of England Density Forecasts of Inflation". Michael P. Clements
The Economic Journal Vol. 114, No. 498 (Oct., 2004), pp. 844-866.
Awesome (+1)!! Thanks for the reference to the wreath product and the Sylov 2-subgroup. Forgetting the last (top) reversion was a mistake, in fact it is included in the scheme.
@Ophelie "I don't already know the groups" Yes but why do you think there are groups? It doesn't look like it. What is your data?
Perhaps this is too pedantic, but to be clear, leave-one-out **IS** k-fold cross validation.  It's a special case, where k=n.
What are you trying to *get* exactly? For most purposes, I'd think that this "bias" is actually a valuable part of the data. But if you want average posts per population, then multiplying by 1/pop seems fine to me. But you might want something else.
Onestop, thanks for answering my question but 0/1 or no/yes does not change the characteristic of the variable. The variable $r_{si}$ is still a binary "categorical" variable. Am I right? Thus, how can a binary categorical variable be divided by a numerical number?
One could almost take these assertions as *definitions* of "continuous." What, then, is *your* definition of a continuous random variable?
Have a look at here: http://stats.stackexchange.com/questions/86019/regression-discontinuity-design-parametric-versus-non-parametric-different-resul
FLD doesn't assume a normal distribution. My question is more about the relations and similarity between FLD and LR and not which one is better. Thanks.
Oooops... Yes, stupid answer. So if the author's question was to calculate the average of k non-repetitive draws, it's also going to be E(X). Leaving this answer just as a programming exercise then :)
It does, though this its use is probably beyond my ken:

http://office.microsoft.com/en-us/excel-help/data-mining-add-ins-HA010342915.aspx
I have a question for you. You said that you are getting "better prediction results". So you are comparing these two models. But this means that you are fitting your Poisson-regression model into two different response variables. One is $log(Y/L)$ and the other one is $log(Y\\times\\sqrt{L})$. In other words, your data-sets are not the same. So I think, it does not make any sense to compare like that.
I went back and accepted a few more answers in my history... To be honest, I can't see the checkmark shades very well and I forget if I've accepted an answer sometimes. Regarding anonymity, why does it matter what my name is if I'm making an effort to contribute content?
Generally, the gist of these kind of problems go: combine and expand the exponents, collect like terms, complete the square, write quadratic as $(x-g(y,\\mu,...))^2+S$, spot the density.
It is results from an experiment but since I am not the one who did the experiment I don't have much information on that. What I am interest is statistically describing the behavior of the data based on whats given to me. Yes. co2 is the reponse and I want to model how co2 depends on density. One tiny thing. co2 is actually co2 rate per minute
This reply starts off really well. The second part, though, does not give a correct CI for a gamma distribution. (Simulate it for small values of `shape` and small sample sizes.)
+1, what a cool answer. Analogy with differential equations seems very apropriate.
Ah, okay. I always had the notion that the parameters for the Dirichlet distribution had to be determined. However, that did not make sense to me since $\\alpha$ and $\\beta$ are set (at least for this version of LDA). Thank you very much!
@Andreea You will have difficulties interacting with this thread because (a) you did not post the question and (b) you do not have enough reputation to comment. A good solution would be for you to re-post this question yourself.  When you do so, please flag this one or the new one for moderator attention so we can combine all the materials into one thread.
In light of your edit, are you interested in testing whether an observed frequency distribution differs from a theoretical distribution?
@cardinal I have to give up for the moment because of external dependencies that I have no time to revolve. I added a pointer to your excellent comment. A lisp/scheme alternative to R has created [some buzz](http://lambda-the-ultimate.org/node/617) some time ago (but see Ross Ihaka's [paper](http://www.stat.auckland.ac.nz/%7Eihaka/downloads/JSM-2010.pdf) for the JSM2010 conference), as you may know; hence my interest for it. But my interest is probably beyond my skills :) I wish I have as precious comments as yours...
A severe problem with smoothing before taking ratios is that the smoothed proportions could easily be invalid (they could exceed 1). Furthermore, the smoothing ought to account for relative uncertainty in the values. For instance, in the series $a=(1000,4,2000,3,3000,2,4000,1)$, $b=(10000,10,10000,10,10000,10,10000,10)$ the ratios are $(.1,.4,.2,.3,.3,.2,.4,.1)$, which would smooth to around a steady $0.25$, but the values alternate in reliability. Using the most reliable values would suggest the smooth ought to rise steadily from $0.1$ to $0.4$.
If you have enough studies, have you looked into using linear meta-regression or conducting meta-regression with dummy coding (e.g. 0 = not being the category you are testing; 1 = category you are testing). The first should expect a significant correlation if the data is ordinal (e.g. mild > moderate > severe), while latter would work better with categorical data (e.g. only correlation with moderate, but not mild or severe).
@gd047: here's a nice reference: http://elevatorlady.ca/doc/refcard/expressions.html
The first question to ask yourself here is *does it matter?*  That can be answered only in a particular analytic or decision context.  What is yours?  What role will this attribute play in your analyses?
@Harvey - Brandon stated in a comment to his Q that he wanted a model for prediction purposes.
I think you're justified to try both, Gower+hierarchical and two-step. Generally, hierarchical clustering is not recommended with more than about 300-500 objects, because it is greedy algorithm and can give suboptimal solutions on later steps if the number of steps is large; but I tentatively suppose this does not pertain to single or complete methods (average -??). Two-step cluster indeed assumes uncorrelated variables, though it is quite robust in this respect. I allows metrical and nominal variables, so you'll probably have to treat your ordinal one as metrical and binary ones as nominal.
Your description is subtly wrong; $\\text{proposal}(\\theta)$ should be $\\text{proposal}(\\theta | \\theta^{can})$ and similarly for $\\text{proposal}(\\theta^{can})$, which should be $\\text{proposal}(\\theta^{can} | \\theta)$.  Try writing out the expressions for the two proposals and finding the ratio, cancelling out all the terms that you can.  And no, $\\text{proposal}(\\theta) \\ne 0$.
+1,This one is more sensible!
How would the units be anything other than 1.0 with your definition?
+1 ... for the answer and comments combined.
Corone's answer below covers the details very nicely. The original figure you have above presents the log-odds values on the x-axis, which can be mathematically transformed to probabilities (i.e. as per Corone's answer, by passing back through the link function.)
The basic measure which you probably should try first is the squared error between prediction and target. That is, you do K-fold cross validation and average the error of each test case. You can then optimize this error with respect to $\\lambda$ to find the lambda which gives the best predictions. I think this is also what your linked talk is suggesting
The same logic can still apply.  A new feature is highly colinnear with a prior feature can still bring more predictive power.  Specifically for a randomForest: if the near duplicate feature is important in general, one version or another is now more likely to be selected as splitting candidates.
@whuber you're right, $B$ and $D$ should be negative. Edited the question.
Do I have to truncate the interval of the explicative variables to study my temperature in the same field before doing the modelization with Gaussian mixture?
Is it more efficient that k-neighbour clusterisation?
The DMZ and crowd 'middle' visualisation is for a classification, whereby the we are, in essence, also visualising an underlying pdf (or combined pdf).
@probabilityslogic, I don't quite understand how to do "linear regression to model the **odds** of being employed vs unemployed for males and females". Could you please either try explaining it using the numbers from my data, or showing me R idioms, or link me to what I should read if you can, or suggest that I ask a new question? As far as theoretical equations go all I understand that in your example beta-0 is our intercept, beta-1 is our slope, x is our data, and e is some error... which is same as to say that I understand nothing. How embarasing, I'm sorry.
No i mean a measure to resume my datas in these particular kind of distribution, like the mean or median for a normal distribution.
It would be a lot easier if, in place of the code, you'd provide the math.
I don't see double seasonalities in your time plot. And since there is only one year of data plotted, it is not possible to estimate the annual seasonality without some very strong assumptions. Use msts if there are multiple seasonal periods and ts if there is only one. And give a reproducible example as requested.
@Elvis According to [Wikipedia](http://en.wikipedia.org/wiki/Dirac_comb), a Dirac comb corresponds to a *periodic* (and therefore infinite) set of data.  Although the machinery is similar, the concept does not appear to apply to this question.
What are the null and alternative hypotheses? What underlying (in nontechnical words) are you interested in answering?
+1 for an interesting question, particularly because you can tolerate a small amount of non-normality which raises lots of interesting points.  But do you have any views on the pros and cons in your particular case of the standard tests - Shapiro-Wilk, Anderson-Darling, etc. - which certainly fit your criterion 1.
@user603 , any thoughts on this? :)