tttjjj/elt_bench huggingface dataset:
In most situations where the concept of $R^2$ is extended beyond classical linear regression, it is the squared correlation between the observed values and those predicted under the model. Could that possibly be applicable here?
Then I definitely wouldn't include components of the score as covariates, for the reason stated above. This older but very useful paper might help further expand your horizons a bit (Harrell's 1996 Stats Med paper on multivariable prognostic models - http://www.unt.edu/rss/class/Jon/MiscDocs/Harrell_1996.pdf)
Thanks.  Do you mean that my general idea of computing each feature value in terms of bigram (N-gram) is correct? In other words, there is no big difference in computing the feature values between bag-of-words and N-gram model. Thanks for the clarification.
What happens when, say, you sum the number 2 to all alphas and subtract 2 to nu? What distribution do you get?
Indeed, the REML estimator of a variance component is usually (approximately) unbiased, while the ML estimator is negatively biased. However, the ML estimator usually has lower mean-squared error (MSE) than the REML estimator. So, if you want to be right on average, go with REML, but you pay for this with larger variability in the estimates. If you want to be closer to the true value on average, go with ML, but you pay for this with negative bias.
I think it cannot be used because it computes the correlation between ranks, so the correlation between `[1,2,3]` and `[2,3,4]`, and the correlation between `[1,2,3]` and `[2,3,5]` is the same (and I don't want it to). Or did I misunderstand?
The derivation of the formula for point biserial r from the Pearson product moment correlation formula is given on pages 29-30 in the text Applied Multiple Regression/Correlation Analysis, 3rd ed., by Cohen, Cohen, West, and Aiken (Lawrence Erlbaum).
I have not checked the algebra in detail, but your derivation looks correct – thank you.  In general, there are two approaches for evaluating a delta method expansion of variance.  One approach (different from yours) is to Taylor expand $g(X)$ in the expression, $Var[g(X)] = E[g(X)-E[g(X)])^{2}]$.
Dear EdM,
Thanks for your reply. Taking the mean for each region will give me an idea about that region's methylation status, but it will not tell me if a specific region is differentially methylated between the two cell lines. For that I need to test each region separately.
The issue about "global" methylation that you address is also an interesting question, but here I am specifically interested in specific regulatory regions. This because differences in methylation for specific regions might indicate differential biological regulation of genes between the samples.
Thanks. Martin
Look up books by Wolfgang Woess as editor or author. A recent collection: Random walks, boundaries and spectra. Birkhauser, 2011. From 2000 (Cambridge Univ.Press): Random walks on infinite graphs and groups.
Good links, they introduced me to the series of `bigX` `R` packages. Looks like `biglm` will do the trick here (http://cran.r-project.org/web/packages/biglm/index.html).
Hm, isn't the main problem the reduction of information you get depending on which treatment design you adhere to? What about if experimental units were randomly assigned to the 10 treatments. The treatment design could then either be considered as 10 separate treatments or as a combination of 2 factors as in a factorial design. Then I could run a ANOVA with either the 10 separate treatments or I could run a ANOVA with the two factors and the interaction, that is to say, the factorial design would yield information regarding the factors and information instead of just the treatments?
@user603 (welcome back :) I've +1, but I still like Greene's book (maybe just because I'm not an econometrician and it highlights another approach compared to that found in psychophysics or psychometrics)
Thanks @jbowman -- in addition you conducted a one sided test, I see. I think this makes more sense then the two sided tests I used. If the normal approximation is done one sided, the p values is .023 above. I like the poit about independence. Probably students also were not isolated when they held their feet into the water, so that's also a way of transmitting a cold.
It seems to me that you systematically replace information that could be obtained from data by assumptions that have little or no objective support, including the assumed probability distribution, that you meet people at random, that names are assigned independently, that you can ignore the J's in the next 992 names, etc.  This kind of exercise can be useful to get a sense of what a solution might look like and what assumptions are important to verify, but it's unrealistic to suppose that the actual chances you obtain can be trusted.
Thanks Alecos, it was very useful following your derivation...
Actors on the inner ring and films on the outer perhaps? Could be an interesting mix, cheers.
@Henry: the intensity distributions are continuous and non-negative.  I'm guessing that for a given population of cells, GFP intensity could be reasonably well approximated by a gamma distribution
This is a very well posed and interesting question.  You might want to distinguish between time series interpolation and other forms of interpolation (such as splining or spatial interpolation), due to the inherent directionality of time series.
I ran your script but swapped `confidence ~ quality` since I'm rather interested in the effect of experienced quality on user confidence (is that the correct notation?). Running your script I get `newdata' had 1 rows but variable(s) found have 4938 rows` four times, and four times the exact same plots output as PDF (note the missing `.pdf` extension in your script). The only thing differing in the plots is the `xlab`, the numbers and histogram (?) are the same.
@John, 1.5 IQR is Tukey's original recommendation for the longest whiskers on a boxplot and 3 IQR is his recommendation for marking points as "far outliers" (a riff on a popular 60's phrase). This is built into many boxplot algorithms. The recommendation is analyzed theoretically in Hoaglin, Mosteller, & Tukey, *Understanding Robust and Exploratory Data Analysis.*
Welcome to the site, @AmaruKuttu. Please only use the "Your Answer" field to provide original answers to the question at the top of the thread. I recognize that not being able to comment initially is frustrating, but as you gain rep, you will be able to shortly.
We have successfully used LimeSurvey with about 4500 participants and a much longer survey (30 pages, 5-15 questions each) with branching and skipping logic. I don't know about hosting though since our installation was at our university's datacententer. Make sure you get fast loading times, bad performance can easily annoy potential participants and drive them off.
Do you need to find the value of the deterministic parameter as well or do you have it avalilable?
Your answer is now almost correct. If you generate $n$ iid $\\mathrm{Exp}(1)$ and divide each by the sum, then you will get the correct distribution. See [Dirichlet distribution](http://en.wikipedia.org/wiki/Dirichlet_distribution) for more details, though it doesn't discuss this *explicitly*.
http://stats.stackexchange.com/questions/tagged/feature-selection
@bayerj: so far I've looked at glmnet and bayesglm.  Bayesglm seems to be noticeably better, but I'm not certain of the correct methodology for variable selection.
This answer is missing a) A reason why ANOVA is not the best idea, b) A reason why mixed model are better and c) any help in finding a reference to them. As such, I don't feel it is a help to the OP.
Colin, an unbiased estimator of the standard deviation does not have a closed form representation in the general case. What does exist is the unbiased estimator of the <i>variance</i> (s<sup>2</sup> in this case).
Noteworthy that both are consistent estimators of the population variance - and so by the continuous mapping theorem, are the two estimators of the standard deviations.
A related point is that s<sub>n</sub><sup>2</sup> has a lower MSE than s<sup>2</sup>. The additional advantage from imposing unbiasedness is arguable.
+1 For those who have not read it, I highly recommend Gould's 1985 [The Median Isn't the Message](http://www.stat.berkeley.edu/users/rice/Stat2/GouldCancer.html) essay in Discover.
Can you post an example of your dataframe so it makes it easier to see what kind of data you have and answer your question.
For two-class LDA, wouldn't Y be just a scalar (or, if you will, a one-dimensional vector)? My best guess is, you may be confusing a dimensionality reduction PCA step (sometimes done before LDA) with LDA itself.
Great answer Jeromy, I will make use of the resources you offered. thanks!
Are you able to post some/all of your data or some representative dummy data?
Accepted the answer, I think I understand this much better now, thanks again.  I edited my question though, if you have time please let me know if I am correct in my understanding.
You need to add a lot more context
Sure. You can drop _any_ items you like and rerun FA. The question is _why_ to do so? To answer, one should be clearly aware of the goal of his analysis. For example, dropping/rerunning may be logical in selecting "good" items for a questionnaire scale. If FA is used in exploration, dropping may be not good think because it impoverish the field of the phenomena under study.
Yes, in the general case this is used for time series with non independent observations; but you can always assume incorrelation between successive observations, which gives the special case of interest to you.
... because you divided by $n$, of course.  The first log likelihood corresponds to a log likelihood associated with a sample of size $n$, the second to a log likelihood associated with a sample of size 1 where the single observed $x$ equals the term you write as ${1 \\over n}\\sum \\log(x_i)$.
@AstrOne - using a symmetric proposal distribution is not sufficient to make the proposal ignorable when calculating the M-H acceptance probability.  Your example shows why.  What you need is $p(\\text{current}|\\text{proposal}) = p(\\text{proposal}|\\text{current})$, i.e., if the ratio doesn't equal 1, you can't ignore them.
Because we're interested in the prob. of hitting one before the other. If we stop on a $7$ we know for sure that we did not hit an $8$. If we stop on an $8$ we know for sure that we can't hit a $7$ without hitting an $8$. If we stop on $6$ we learn nothing about the probability of hitting a $7$ before hitting an $8$.
Can you clarify what you mean by "non-numeric" variables? Categorical (ordered?)?
It is not. My answer is just a first step towards a direct discretization of the beta distribution, so it remains to be checked and studied, for results, moments, properties to be derived for it etc. On the other hand, the Beta-Binomial is an already well-studied distribution. On a first check, they seem to provide roughly analogous results regarding the shape of the pmf graph.
@Dail I've updated my answer for functions that implement the White-Test and the heteroscedasticity-consistent standard errors.
I wonder why nobody has mentioned the "quasi-poisson" option? was that already implied ?
Oh my model was lm( yVar^2 ~ xVar+xVar+xVar ), i transformed it by squaring because otherwise the residuals weren't normally distributed. The Y^2 label is a bit confusing though as it seems like the Y should be the transformed Y already.
You should post your code here.
If you have Likert items, you cannot have true normality. Maximum likelihood, etc, are procedures for estimating the model once given the correlation matrix; the issue w/ likert data vs other types has to do w/ how to get the appropriate cor matrix, not the estimator to use subsequently. You would benefit from taking some classes or reading some textbooks on the topic.
Here is a post with the code and some illustrations:

http://www.markmfredrickson.com/thoughts/2010-08-06-combinadics-in-r.html
@PatrickCoulombe Thanks for your comment. The problem here is that 1. This problem is sort of well-known, as the OP guessed, and 2. There is A LOT of notation involved, which would require an afternoon of typing. I will add some references at the end, just in case the link is sent to Limbo. A basic google search returns several documents with the posterior of interest.
@DilipSarwate You didn't read the question carefully, did you? Quote: "I can't see how $\\mathbb{E}[X^{T}AX] = tr(\\mathbb{E}[X^{T}AX])$. I think I get the rest of the proof [...]" The part Kyle was confused about was exactly the part I addressed.
If you're studying all lapsed donors, power analysis is a bit irrelevant (I'd have though), because you're going to do them all, and you can't get a bigger sample, even if you want to.  I also agree that the population of interest is all lapsed donors in the future, and so even though you are, in a sense, including the whole population that is available to you, you are not using the whole population and therefore you should do inferential statistics.
An explanation of what this code is doing would be welcome.
Welcome to our community, Georgette!
are these all different subjects? i.e. did you ask different genre questions to the same person in this study, and if you did is there any way to link the subjects? basically, imagine that in an extreme case there were only two people: one man and one woman. it's an important detail
@JeremyMiles Of course you can have a directional hypothesis with three groups in ANOVA. e.g. with three groups, you certainly can have an alternative like "$\\mu_b \\leq \\mu_c \\leq \\mu_a, \\text{with at least one inequality being strict}$". The difficulty, if anything, isn't so much that they don't exist, it's that there are *so many of them* - with $k$ groups there are $k!$ such fully specified directional alternatives. It's just not a straight $F$ test. There are both parametric and nonparametric analyses for these situations.
**Friendly reminder**: The more effort you put into stating your question clearly and completely, the more easily others can provide clear and complete answers. :)
+1 for anomaly detection. I'd also add "number of attempted logins last 5 minutes" and "number of attempted logins from ip X last 5 minutes".
Thanks for the suggestion.  This function is certainly useful, though my interest is more in terms of statistical practices and methodology rather than code for a particular solution.  This is akin to asking about sampling methods or reproducible analyses, rather than particular packages that implement said methods and analyses.  Nonetheless, good methods should become packages.  :)
Hmm the t-test also assumes a normal distribution. Perhaps what you meant to say is that we require less information about that distribution.
@shabbychef: Java programs doesn't have to be a slow monsters. I admit that well written C-code will almost always be faster than a similar implementation in Java but well written Java code will most likely not be super slow. Plus developing in Java has many significant advantages.
+1 for mentioning a survival analysis approach.
There's [PSPP](http://www.gnu.org/software/pspp/).
Thanks for the note. I updated my question to include the priors on beta_0 and tau; I had omitted these for simplicity
"95% CI of 19" --- confidence intervals are not points, but ranges; you'd normally specify at least two numbers for it to be a CI. Is this a one-sided CI? Or did you mean the width of the interval is 19, or something else?
Thanks for sharing your point of view. Given reference is really informative. Thanks again.
Never thought about such use of program statistics. My respect to you!
To my eye, it's not clear how you'd mathematically define the minima points along your red line. You know what you want, by eye, but I can't see a clear and precise definition. You could take your standard regression and then subtract some value of the SD from it. Or do a quantile regression with a small quantile number (at some point, it will break, so you can't just use 0 or 0.01).
nice, i just finished reading the drunk and dog paper as i saw your comment. its got a nice example and its a well written introduction to get me started. thank you for all these references! im just an undergraduate but i feel like im well on my way to a phd in statistics. lol
take the diagonal of `S`, if it is not already a diagonal, square it, sort it in decreasing order, take the cumulative sum, divide by the last value, then plot it.
Failure to reject a hypothesis test (for a null of independence) doesn't tell you that residuals *are* independent.
Maybe it's me, but I don't understand what you  mean by "smooth values on a graph". Which values do you want to smooth? Usually smoothing refers to a distribution, what is distributed here?
Thanks everyone for your responses.  Concerning several comments, whether a lead becomes an opportunity is a definite yes/no opportunity.  Leads that don't convert to an opportunity are eventually marked as dead.  Also, I've done a little more research and see the error of my ways with the use of stepwise selection.  Seriously, thanks everyone for taking the time to help out a newbie.
+1 Mixture models are useful. Especially if you have data that is generated under two or more different circumstances.
Thanks for the help.  This is where show my novice.  I , simply have been working through the basic examples in the r packages and aggregated the data to a point where I created my own graph.  Up to this point, it's been an exercise to get started in network analysis and data visualization.  This is a huge step forward though, thanks.
Significance of what? What hypothesis do you wish to test?
No, my point is that logistic regression et al do not "work" in the sense I'm describing with both types of data.  They require you to, at least in some sense, treat all predictors as numbers or none of them as numbers.  I know, for instance, that one can often get great results with a logistic regression by coding something like "gender" as 1 for "male" and 0 for "female."  However, I'm wondering whether this type of process can be avoided with any known modeling paradigm.
Stepwise model selection is as much data dredging as full subset selection (it actually tries to find approximatly the same solution in much less time). AIC-based selection is data dredging too.
No, quasipoisson should not be different; the only difference between quasi-Poisson and Poisson is the estimate of dispersion.
Terminology suggestion: when using PCA, always refer to principal components (PCs), and never factors.
If I multiplied the probabilities in method 1 by 100, would the "unit increase" result in the same interpretation when comparing the OR of the two methods? There is still the issue that one is disrecete and one is continuous, but at least the unit increase is 1 in both cases, not 0.1. The range would still be very different, and a 5 from method 1 means something very different in Method 2 (in terms of severity), how much does that matter?
If $\\mathbb{E} [Y_t | X_t] = X_t$, we have $Y_t = X_t + Z_t$, where $Z_t$ is a random variable  with a density symmetric about zero. 

So, $\\mathbb{Var}Y_t = \\mathbb{Var} X_t + \\mathbb{Var} Z_t + 2 \\mathbb{Cov} (X_t, Z_t)$

$\\implies \\mathbb{E}Y_t^2 = \\mathbb{E} X_t^2 + \\mathbb{Var} Z_t + 2 \\mathbb{Cov} (X_t, Z_t)$

Therefore, I don't know how one can obtain a reasonable estimate of $\\mathbb{E} X_t^2$ without making some assumptions about $Z_t$.

Do you know anything about the conditional distribution of $Y_t$ given $X_t$?
Thanks for the elaboration. I'm not just interested in the MLE, though. Since this is Gibbs sampling, I need a random mu sampled from the likelihood of mu given fixed data/sigma.
I think if somebody proves that a sample correlation itself is an unbiased estimate for a measurement error, there should be no problem in making an inference that we can work without using such a correction. To me, it seems that sample correlation is unbiased for measurement error because it is a quantity that uses SD values in the process.
Completely agree with you. I have a bachelors in computer science, so I really like to code my statistics. Unfortunately, time is scarce since I'm in the finishing stages of my dissertation, but once I have more time I definetely want to learn more about OpenMX, which seems to give people more flexibility.
@jkd Unfortunately no - simulating it on a network, and the effect of the network on the hazard is unknown.
The estimation is the same for ordinal "X" categorical variables as simple categorical ones - the interpretation is different though.  The fact that they are ordinal categories (a,b,c,d,e) simply tell you that a<b<c<d<e and you would need some structure saying in some way "how much" bigger each category is compared to the ones below it.  If you want to save a few degrees of freedom, then you would need to specify some additional structure saying how the ordering happens using 3 parameters or less.
@4everlearning Right, after taking seasonal differences the ACF and PACF looks closer to what we could expect for a stationary process. You can start by fitting an ARIMA(2,0,0)(0,1,1), in R `arima(x, order = c(2,0,0), seasonal = list(order = c(0,1,1)))`, and displaying the ACF and PACF of the residuals. Be also aware of further issues raised by IrishStat that you should be concerned with in the analysis.
Excellent, thank you. I see there's a Wikipedia article on the Cauchy-Schwartz inequality which goes into the proof: http://en.wikipedia.org/wiki/Cauchy-Schwartz_inequality
An example with the above data, for the record: mod1 <- glm(Days~Age+Sex, data=quine, family="poisson"); if(interactive()) {vis.test(residuals(mod1, type="response"), vt.qqnorm, nrow=5, ncol=5, npage=3)}
It's difficult to talk about practical solutions for situations I don't know anything about - when I do practical analysis, it's in the context of knowing (i) what the data *are* - the variables and what they mean, how they might be expected to behave; (ii) what questions I need to answer, *under what assumptions* and how strongly I might believe them *a priori*; and (importantly) (iii) the data values themselves. In the absence of reasonably detailed context, I can't really say much more about practicalities, but I bet at some point I'd estimate a common lag 1 autocorrelation.
great answer.Thanks..
Tip: next time use the button with binary numbers (or indent it with 4 spaces) to make the markdown engine interpret the code as code.
@Casey - sorry I misread your requirements re zero truncation. Shame that geepack doesn't work with that family function. If I think of anything else, I'll update here.
Thanks, I haven't seen many of these in the wild, but using a panel is a great way to accomplish this!
I think the fourth equation is incorrect.  We should have $P(D_iD_jH_b|X)=P(D_iH_B|X)P(D_j|H_bX)$
It's easier and more efficient simply to draw one value from the data at random.
First, thanks very much for your reply, and for the summarisation of my ideas in a well-structured way (using proper terminology). I should say here that I am **not required** to use any particular method, rather the question was in regard to a recent suggestion that I received - to try incorporate the approach presented in the paper (after the introduced two algorithms) to my particular problem, explained above. Hope that clarifies things a bit.
@steadyfish, You can derive the regression equations by either assuming the x's are fixed values and the y's are random based on the regression equation, or you can assume that x and y follow a multivariate normal distribution.  Either way leads to the same set of regression equations so we generally don't distinguish between the cases and just treat the x's as fixed.  Some purists may see a philosophical difference between the 2 approaches however (and which bootstrap method to use could depend on which way you think of the data).
@LadislavNado That's my reference to meta-regression. It *may* be a solution, but is not guaranteed to be.