tttjjj/elt_bench huggingface dataset:
Emission probabilities are used in **hidden** Markov models and refer to the probability of an observation given a hidden state. Generally you have hidden state $X_i$ which is related to $X_{i-1}$ by **transition** probabilities and an observation $Y_i$ which is related to $X_i$ by **emission** probabilities. The $X_i$ are never observed and need to be inferred using Vitterbi algorithm or expectation maximisation if the transition probabilities also unknown.
I edited my answer with some remarks on that.
The wikipedia page [gives an algorithm](http://en.wikipedia.org/wiki/Normal-gamma_distribution#Generating_normal-gamma_random_variates)
I think your whole derivation is based on assuming that $y$ is positive. Maybe break it into cases based on the sign of $y$ and go from there? Also, it's hard to follow your argument in a straight line of text like it is in your comment - I suggest editing your answer, instead of posting modifications in comment form, to aide understanding.
I would just like to mention that as someone interested in making generic MCMC algorithms, easily applicable to many problems, I am very interested in this topic.
@cardinal -- I haven't seen either (maybe I shouldn't have converted to var/covar -- I thought it might save an answerer some time). I am more familiar with $R^2 = 1 - SS_{err}/SS_{tot}$ version. Why does the var/covar form work?
@MånsT: I am look for "all unbiased estimators of zero".  I don't understand that question.  And also, what is $\\theta$ here?  I edited the question.
Welcome to the site, @JoeSullivan. I took the liberty of using $\\LaTeX$ to make your equations easier to read. Please make sure they still say what you want.
+1. I really like the idea to compare the QQ-plots of your sample with some randomly generated ones!
The discussion in the help `?anova.glm` looks like it is relevant to your question (though not completely covering the 'why').
I agree, Markowitz definitely invented portfolio theory
+1, nice answer. Of course there's still an integral hiding in there ;)
@Michael, I agree with you. In these kinds of hierarchical models, the random effects are defined by a grouping variable (as opposed to other multivariate models such as spatially indexed data sets, where the 'grouping' variable is continuously varying). In the OP's question, `Site` would be referred to as the random effect, not `T` or `A` or anything else. Thinking of it that way, `Site`'s effect clearly could not be both fixed and random, since the two wouldn't be identified from each other. You can have both fixed and random coefficients for a variable, but that's a different question.
@ConjugatePrior: I understand things much better now. Thanks for your excellent answer!
This one's google-fu is strong
(+1) I would just add that the argument for $x+y>\\epsilon\\Rightarrow (x>\\epsilon/2 \\;\\mathrm{or}\\; y>\\epsilon/2)$ is more specifically named an argument by contraposition (see http://en.wikipedia.org/wiki/Contraposition).
@Andy Amos. Can you please provide some more details. At this point, I can only say -- probably -- hierarchical Bayesian models would work. cf. Gelman and Hill 2007, Cambridge Univ. Press
In experiment #2, what is the chance a given ball is drawn exactly once? I am thinking it is $(2k)(1/n)(1-[1/n])^{2k-1}.$ Isn't this what you want to maximize?
I don't know what is AB testing but this thread has been automatically linked to yours: http://stats.stackexchange.com/questions/12739/a-b-testing-in-python-or-r
As a whole is non-normal for Area and Roundness but it's normal within the group of replicates for all the three predictors.
No, I mean [this distribution](http://en.wikipedia.org/wiki/Multinomial_distribution)
Haven't seen it either. Not sure why it would help students to have yet another apparently arbitrary formula to remember.
50% missing values for a crucial variable? Ugh. Rather than impute, why not create a 'Missing' category for the variable?
I think it would actually be $\\sqrt{1-\\rho^{2}} \\frac{X_1}{X_2}+\\rho = \\sqrt{1-\\rho^{2}}Z+\\rho$ from $$\\frac{\\sigma_1}{\\sigma_2} \\left[\\frac{\\sqrt{1-\\rho^{2}} \\left(\\frac{\\mu_1}{\\sigma_1}+Y_1 \\right)+\\rho \\left(\\frac{\\mu_2}{\\sigma_2}+Y_2 \\right)}{\\frac{\\mu_2}{\\sigma_2}+Y_2} \\right]$$
Details [here](http://stats.stackexchange.com/questions/56230/modelling-two-correlated-variables/56336#56336)
If you mean reading the forest by looking at each tree and trying to understand what is going on, don't do it, you'll fail to see anything in such a complex, randomized structure. RF is just a black box method, its models are not intended to describe data (as usual in machine learning).
This is a functional time series question. See literature by Hyndman. I vote to re-open this question.
For some perspective: If the model explains nothing, so that the coefficients are essentially randomly distributed around zero, then there is a one in eight chance that all four coefficients have the same sign (one in 16, including the intercept). Such chances are not in the least unusual.
@AdamO Because it destroys the *within-cluster* correlation.  Maintaining that correlation is the logic behind sampling the entire cluster.
How large of a population effect do you want to be able to detect or how wide do you want your interval to be?
Are you sure he has enough data points to produce meaningful spectral analysis? Though I also do like to work in the frequency domain from time to time :) (+1) for the link though, will add to my collection of 'yet to read' things.
@whuber I meant the distributions are given; I only wanted help with the probability problem.
You don't have independence of sucessive points for the relationship between "time so far" and "% completed so far". You can't ignore that. But in any case I don't think linear regression is the tool you want.
+1: this question usually exposes some interesting point of views.
@MichaelChernick Note that wasn't a criticism **from** me, just offering a reason why people might have down voted. I would disagree with them if that was the reason because I think your answer is spot on, especially with PRIM; I was just consulting my [Hastie et al (2009)](http://www-stat.stanford.edu/~tibs/ElemStatLearn/) as to what it said on PRIM. You might want to add that link to the Answer as there are two sections on PRIM there and the PDF is available for free.
Welcome to the site. If you'd like to edit your post and add a link to the picture you want displayed, then I or someone else can follow-up with an additional edit to actually show it. Cheers. :)
Ahh I see. So what are the equations we should be fitting in a pure interaction model? Both R and SAS are giving me the same results...
Thanks for the answer. I will look into them
53 characteristic variables? These sound like your independent variables (of which some are collinear as you've indicated) while `col1` and `col2` sound like your groups. What you're looking for is probably multiple linear regression modelling, and definitely some rigorous model selection to check for multicollinearity (to remove bias) and find significant predictors (using AIC stepwise selections). Check out [this link](http://www.statmethods.net/stats/regression.html) for some example code in R.
Yes, I completely agree it's Monte Carlo; I already said it was earlier, and I agreed it contains a Markov Chain - so quoting the definition of Monte Carlo or Markov Chain to me achieves nothing, since those aspects of what you're doing are not in dispute. I already explained, MCMC is not just "simulating a Markov Chain". Since you're avidly checking wikipedia, I have to wonder why you don't instead quote the first sentence of the article on *Markov chain Monte Carlo*. The last half of that sentence gives a crucial difference, but it may not be clear what it's getting at.
Section 2.2 of Gerald van Belle's book, "Statistical Rules of Thumb" is called "Calculating Sample Size Using the Coefficient of Variation".  This is in the context of knowing only the CV and the % change in the mean you expect between test and control, and a standard power. The sample size (per group) = 16(CV^2) / (ln(mean1)-ln(mean2)^2.   So if the CV is 30% and the expected difference in means is 20% each group should have >=29 in it.  This is the two sample case, and you're asking about one sample, so some further algebra is needed.
Thank you. This may be, as you say, not a complete answer but I will mark it as accepted since this is the closest I think I'll get. I'll study it and see if I can make it work for me.
Have a look at my question here: http://stats.stackexchange.com/questions/1881/analysis-of-cross-correlation-between-point-processes
Where do the "25"s come from?  Do you have references or can you explain the theory that justifies this proposal?
That's unfortunately one of Protovis's weak points (but it's an issue with any SVG library because IE doesn't support that).  Fortunately, Jamie Love has come up with a solution using SVGWeb.  See here: http://groups.google.com/group/protovis/browse_thread/thread/1a80f98a16736658?pli=1.
It looks like you may have mixed up the meanings of *goodness of fit* and *effect size.* None of these criteria address "how ... steep" the slope is, as requested in the question.
It is actually possible to use compound symmetry with `lme` to obtain the same results as with `aov` (and thereby enabling `lme` for all ANOVAs), namely using the correlation argument in the call to `lme`: `anova(lme(value ~ factor, data = tau.base, random = ~1|id, correlation = corCompSymm(form = ~1|id)))`
From the statistical point of view, perhaps. But it's usually a nightmare for interpretation, because such removal can cause the reference to be overly heterogeneous. For example, if we wish to compare different race/ethnicity using "white" as reference, and then blindly aggregate some other non-significant categories (e.g. "Asian" or "Hispanic") into "white", then the original research question can no longer be answered. To aggregate or not shouldn't be only based on p-values, contexts and hypotheses matter as well.
(+1) is there an `R` package that makes these plots (I recall whuber displaying a plot like this once) or did you do this from scratch?
It's not always pointless, Bill: it depends on the outcome. If, for instance, that cyan blob were far to the right of the red blob, we could conclude the two are significantly different.  That's the whole point to testing: it helps us decide whether two estimates should be considered non-overlapping ("significant") or whether it's just too difficult to tell ("fail to reject the null hypothesis"). The amount of fuzziness, or horizontal spread, depends on the amount of independent data available and how much they vary: lots of fuzziness does not necessarily imply the model is poor.
@gung Real life intervened so I posted a partial answer.  Updated.
@Glen_b But that's not what we're testing! It's *assumed* $\\mathbb{E}(g)=0.$ (Actually it's fine if $\\mathbb{E}(g)=\\mu$ is constant.) We don't care at all about the size of $g$; it's merely supplying additive error to the set of second differences of $f$ that I described. Those second differences equal $g(x_1,y_1)+g(x_2,y_2)-g(x_1,y_2)-g(x_2,y_1)$ under the null, enabling us to turn the data into a (very large) set of values viewed as the realization of a (very large) multivariate distribution with zero means and *known* covariance matrix, up to a constant: that's the setting for a t-test.
I agree to what you've said. But the approach I use currently is giving me promising results. I predicted 20 matches and it gave me a 70% accuracy. 
I'll still give a look to regularizing Naive Bayes.
Wikipedia has some good content on this; see http://en.wikipedia.org/wiki/Pareto_distribution#Relation_to_the_exponential_distribution.
It would make answering easier if you specified how did you do your calculations.
No, the minimum sample size is pretty much unaffected by the population size, unless the population is quite small.
More than one thing can have a chi-square distribution. The thing you link to is not exactly the more general thing you're asking abut at the start, though *in the particular case of multinomial or Poisson $y$* and under the right conditioning, they will end up the same.
Interesting... I'm a bit busy at the moment, but I'll definitely have a look at that!
WHen the denominator df in an F go to infinity, it's exactly equivalent to a chi-square
First you need to tell us what you mean by "variability". Any answer to your questions could be correct.
There are "several" algorithms that the Prediction API can choose from when training/predicting your data.  The engine chooses the one that it decides is best.  

Some users have requested a little more control over that selection, http://goo.gl/mod/5EoA, even if the algorithm is unknown.  

Redditors have speculated on the guts here, http://www.reddit.com/r/MachineLearning/comments/evdxb/what_are_your_thoughts_on_google_prediction_api/, but the stat-speak is lost on me.
What formula are you using to compute that linear separator?  (I presume you are regressing the symbol--which has only two classes--against the $x$ and $y$ coordinates in the plot and then using the estimated coefficients to identify the line.)
It's normal to get dull and trivial results instead of something "Wow, interesting!". But of course it's your right to pre-process data in a way you like. For example, you could log-transform variables such as money (income?)
Sorry, I missed that you were talking about explanatory variables. I've deleted the useless comment.
Wow, Thank you very much. It really is magnificent! Matrix form really pays off! In summary, SSE/$\\sigma^2 = Y^T(I-H)Y$ and $I-H$ is idempotent. Idempotent matrices have eigenvalues either 0 or 1. So sum of eigenvalues is the number of eigenvalue 1. and $tr(I_n-H)= tr(I_n)-tr(H)=tr(I_n)-tr(X(X^T X)^-1 X^T)=tr(I_n)-tr((X^T X)^-1 X^T X)$ since $tr(AB)=tr(BA)$, and $tr(I_n-H)$ becomes n-p+1. and sum of eigenvalues of a matrix is sum of traces of the matrix! and $I-H$ can be expressed as $ADA^T$. So the first $Y^T(I-H)Y$ becomes $Y^TADA^TY$ with D with only n-p-1 diagonal 1's.
@Macro No anger intended. I thinkthe OP and I are getting along fine. My caplock was stuck and instead of erasing and retyping I thought the caps would be good for emphasis. Didn't mean to portray anger. I would edit it now if I could.
@jem77bfp: how did you get that?
@gd047 All right, as usual it's always when we look for old code that we cannot found it (despite my best effort with grep/find), so I rewrote a quick (still ugly) [R script](https://gist.github.com/736212) for that. I've also put an example of the [MP code](https://gist.github.com/736216).
If you do a paired t-test, it will be the normality of the pair differences that matter, rather than the raw values.
I think it really has to do with the characteristics of your data--I'm not sure I'd agree with a blanket statement like "tfidf is always worse than binary" either!
thank you zhi zhao for highlighting the easier way to solve this equation :D
The sparsity of SVMs is often rather limited, in my experience, if you choose the hyper-parameters to minimise the cross-validation error, you often end up with an SVM where almost everything is an SV.  If you need sparsity, it is better to use an algorithm where sparsity was a deliberate aim, rather than a happy accident.  Also it is possible to get a sparse ANN model, I used to use the Laplace prior introduced by Peter Williams quite a lot and it generally works fairly well see http://www.mitpressjournals.org/doi/abs/10.1162/neco.1995.7.1.117 .
@cbeleites do you mean http://www.stats.ox.ac.uk/pub/MASS4/ ?
@Federico If in doubt, I'd go with the factorial ANOVA. That said, the factorial ANOVA would give you an estimate of any repetition effects.
This is perfect. I got it now. Thank you
The p-value I get with LRT is 0.04, which is significant, but very close to the threshold of 0.05, so someone can doubt the result. The AIC of the full model is lower, and this is a cleaner result. Actually, AIC of all models that include that one variable in question is lower.
I don't know an exact answer to your question, but in general AIC and BIC can be used in many types of models, also those that are not linear and used to model non-normal dependent variables. So I think the normality assumption is not central.
(+1) for the link and reference. Since the link is to a blog that discusses the `confint` function in R, I want to mention that profiling in R (based on `confint.profile.glm` from the MASS package) is a combination of evaluation of the profile likelihood and a spline interpolation.
Could you give more detail about what you're doing? I'm having a hard time understanding how you are using this weighted average.
I think it would be ideal if you could present some of the data you finally have. At least for me, that would help a lot in thinking about an answer.
see here: http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29#K-fold_cross-validation    k-fold (as opposed to n-fold) cross-validation saves time because now you have to run your procedure only $k$ times, here $k=20$.
@MichaelChernick, if you want to get a user's attention, consider using the '@' symbol immediately before their name so that they will be notified that they were mentioned (as I did with your name here).
My answer would be pretty similar to Wayne's. I always use Python to parse through data. And I use R for statistical analyses, graphs, inference, etc. This has served me well. Just make sure you learn some shell scripting so it is easier to link the two. Technically you can do statistical analysis, graphing, etc. in Python, too, but the community support isn't nearly as strong in those areas as it is in R.
It is unclear what "The IV is measured slightly differently across studies" means (can you give an example, how many categories, etc.). Can you dichotomize your IV? What about conducting a 2-stage analysis and then running a meta-regression to test if different measurements of your IV affect the effect sizes?
Well, I know there are methods to convert the scores (distances from boundary) to probabilities and that it might be more interpretable, however, as it is clearly stated in the question: I want to use these scores "without converting these scores to probabilities". So my question is: does it make sense to take the absolute values of all these scores and say that the higher is the absolute value of a score $|s_i|$ the most likely $x$ is of class $i$ ? Or should I compare the scores without the absolute values ? or ... ?
@EllenMarshall, not necessarily a consequence. The error plot resembles the data distribution has nothing to do with if the variables or model are correctly specified. Generally, time-series data violates the assumption of independence between observations. Some forms of correction will be needed so that your standard error will not appear to be too small.
Pedantry alert: If your measure is symmetric, i.e., $d(a, b) = d(b, a)$ then your measure violates the triangle inequality, since $6 = d(b,a) + d(a,c) < d(b, c) = 8$, and isn't actually a valid distance, which can cause problems.
Neither, we are learning A. Weight can be thought of as the entries of x being weighted by the columns of A. In unsupervised learning we are in some sense learning x. Think of clustering. We are saying what can we say about the structure of x so that some x values are closer than others
So in the ordinal case, can I say that the distance between each class, or the difference between two thresholds (at $i+1$ and $i$), corresponds to the range of probabilities of $class = class_i$?
@chl has nicely answered your question in his reply at http://stats.stackexchange.com/questions/8237/logic-behind-the-anova-f-test-in-simple-linear-regression/8247#8247 : see the second paragraph.
Of course this proof is valid *in the generic case*. But it may happen that for some $x$, $X_2=y$ never happens, then $[W_y=1]=\\varnothing$ hence, with respect to $\\Pr_x$, the event $[W_y=1]$ does belong to $\\mathcal F_1$.
I will also try looking at the scatter plots - thank you for suggesting that approach
I'm restricted to unvariate t-tests.
What specific "difficulty" is arising in regressing these data? (It is not going to be due to the theoretical bounds, because your data come nowhere near them.  It would likely be a mistake to use a regression method, such as Beta regression, that assumes there are bounds *and you estimate those bounds from the data themselves.*)
I have made the distinction between the population means and the resulting sample means more clear
Unfortunately, I do not have the code here at the moment. I basically follow the Wikipedia entry at http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test until I am left with the number $D_n$, which one, as far as I understand, usually looks up in a table to accept or reject the distribution.
Is this restatement correct: You have one measure per day for 60 days and want to predict declines later in the 60 days based on measures earlier in the 60 days. (I do not know what you mean by aberrant measures.)  You may want to get change measures, perhaps 59 day-to-day change measures.  Then you can see if these change measures predict later (larger) changes that are of interest to you. If variance is large, you might look at changes in week to week averages for each participant.  Does that make any sense in light of your specific project?
I find quite confusing the link between "reach a certain point" and "avoid that stop" and "missed a stop or not".  I don't think the problem is statistical jargon, could you maybe re-word the problem more precisely about what in the real world it is about?
How many classes are there?  If there are substantially fewer than $10^4$, then why not estimate all the probabilities directly from the data as $\\hat{P}(A>B)$ = # cases where A is preferred to B / # cases where A is compared to B?  (This is ML and it's as simple as they get.)
Thank you for your answer!
Now I understand the difference between vs within, and my experiment is actually a between-subject design: I have 5 genotypes, each one tested in 3 different treatments, with 5 replicates, so a total of 75 plants.
Still, I don't understand what I should include in "wid"... Is it just a sequence of numbers, from 1 to 75, so each plant has it's own number? If this is the case, what is it for? It doesn't influence the calculations at all, does it?
Thank you very much Stephan!!!
should this be $A = X + \\epsilon Y$ not $Z + \\epsilon Y$?
Here is a link to a fuller list of Excel charting possibilites
http://spreadsheetpage.com/index.php/tip/chart_trendline_formulas/
That this was selected indicates a biased prior.
And to be clear, what I want to test is that the sample I have (the positions and distance between them) is not random position in genome, they are more clustered compared to random positions in genome. So the theoretical values in QQ plot is the distance I got by randomly draw positions from whole genome.
Another one of hadisanji's homework problems, showing no personal attempts at solving whatsoever, to which we are asked to provide complete answers that will rarely ever be accepted?
And intuitive too!
Hmm, I think that part of the problem is that i am not sure what exactly what the model would be in notation.
 
For an observation level predictor X, a repeated measure group level predictor Z, 
and for i observation, j individuals, and g groups, 
Although i am not sure, I think that what i need is something like this:

$\\ Y_{ijg}= \\beta_{0j}+\\beta_{1g}+\\beta_{2j}time_{ij}+ \\beta_{3g}time_{ig}+ \\beta_{4}X_{ij}+e_{ij} $ 

$\\ \\beta_{0j} = \\gamma_{00}+u_{0j} $   

$\\ \\beta_{1g} = \\gamma_{10}+\\gamma_{11}Z_{g}+u_{0g} $ 

$\\ \\beta_{2j} = \\gamma_{20}+u_{2j} $ 

$\\ \\beta_{3g} = \\gamma_{30}+u_{3g} $
Thanks for that. I was thinking in a kind of stepwise regression manner, where you would put age (in my example) in the first step, and then continue with the variables you are focussing on in the next step.