tttjjj/elt_bench huggingface dataset:
@Sonya Yours is a good summary. The key is that when estimating the parameters for the conditional mean, the estimating functions (say the score function) for poisson and quasipoisson are the same! It therefore does not matter for these parameters whether there is a penalization or not as long as it is the same penalization. I make that more clear above. Thanks also for the pointer regarding summary(modq), but that is already there, it just gets "boxed" on a normal screen, so one has to scroll down.
@cardinal For reference a paper on "The Exponentiated Generalized Class of Distributions" Cordeiro et. al is [here](http://www.jds-online.com/file_download/377/JDS-1086.pdf) see page 2, 4th paragraph last three lines.
It looks like you have several observations at each level of the independent variables, in which case you could use weighted least squares to correct for the violation of heteroskedasticity.
Welcome to the site, @AdityaKumarSingh. I don't quite follow your answer. It seems like you are talking about both using the inverse distribution function to output a variate given a probability, & talking about an accept / reject algorithm, as though they are part of the same process. I wonder if you can clarify your answer. Also, since you are new to the site, be sure to read our [FAQ](http://stats.stackexchange.com/faq).
Maybe this could give you some ideas: http://www.ted.com/talks/scott_rickard_the_beautiful_math_behind_the_ugliest_music.html
Sure, I can have a go, though recall that your original comment asked me to *name* approaches... and that's what its name is. In the meantime, there's a good explanation of the basic process [here](http://www.robertowor.com/csci4151/lecture3.htm). It's also called the [Robin Hood method](http://www.jstatsoft.org/v11/i03/paper).
Could I add a Complexity ? What could I do if the legislated values are expressed with mean value and SD ?
"but it will improve reliability of the model out of sample" - this is incorrect, it is very easy to make generalisation performance worse by over-fitting the choice of features.  Also regularisation is what makes the SVM a good tool, it doesn't actually work any better than ridge regression.  The last piece of advice is very dangerous, you need to perform the feature selection separately in each fold of the cross-validation to avoid bias (http://www.pnas.org/content/99/10/6562.abstract).
does this help?  http://stackoverflow.com/questions/14684437/what-does-the-integer-while-setting-the-seed-mean   Also ?set.seed() within R provides pretty good information.
@slava See my edit, it looks like studying variation of microarray data is a field unto itself. Perhaps even if the first results are unreliable, comparing the technical replicates can be a paper on its own.
Rather a non-implemented feature, still you can send a bug report.
@whuber: The data is from a set of NMR sample spectra. The spectra are plots of response over time. The timepoints are taken at the centre of each response peak. Ideally, samples with the same compounds will have peaks at the same timepoints. But usually the actual timepoints of identical compounds can vary a little. That's what I mean by "functionally equivalent". The rows are sorted ascending and when the eventual clusters are sorted, the original order of each sample must be maintained. NA is padding in the matrix, but means that there is no peak in that sample at that time in the df.
Report to whom? For what purpose? If at first you thought it might be important but found it to be negligible, that would seem like a worthwhile thing to report in many circumstances. The only evidence, as opposed to supposition, that it's unreal, & not just unimportant, might come from the smaller sample - did that in fact disconfirm an effect of the size you saw in the larger one, or just fail to find a "significant" effect"?
Even if it does need to be adaptable, put some EXTREME limits in there anyway. If a hardware glitch gives you 1e280 as a value, or NaN or +Inf, you may want to filter those out regardless.
I think he means that the sample average would be an estimate of the expectation for that die.  My problem understanding this is with the term set size.  I am not clear as to why you want to define certain sets and determine the distribution of the size of the set. If the rolls are independent the resulting sample has a multinomial distribution and the sample proportions give estimates of the multinomials parameters.  So in large samples you will have an idea as to whether or not the die is biased and if so which numbers have the higher probabilities of occurrence.
I saw your edit. but  $c := \\int g_\\theta(t) dt$ not $c := \\int g_\\theta(T(x)) dx$. (sorry for mistake in my original post)
(+1) Good high-level description.
the `reshape` package can be useful as well. Once you have `e`, consider something like `library(reshape) cast(melt(e), name1 ~ name2)`
Hey, thanks for the feedback--I was worried that it might come off as sounding condescending and would totally miss the mark. Oh, by the way, an **OLS** regression is basically just your run-of-the-mill regression (i.e. it's the technical name for *the* regression you run in Stats 101). Also, you might find [Think Stats](http://www.greenteapress.com/thinkstats/) helpful considering your programming background (a free pdf is available via that link).
You can have constant coefficient of variation while variance changes. You can even have constant variance while coefficient of variation changes. So looking at (say) Levene's test won't directly tell you about coefficient of variation. Are you asking instead whether those tests might be modified to deal with comparisons of coefficient of variation?
The problem is explained [here](http://www.mail-archive.com/r-help@r-project.org/msg80580.html). For a workaround, you can use `anova()` as described in [this answer](http://stats.stackexchange.com/questions/6865/split-plot-anova-model-comparison-tests-in-r/14207#14207).
The issue of "which term **is** more commonly used" can easily be addressed, albeit crudely: A Google search of "Gaussian" distribution has about 2/3 of the hits of a search for "normal distribution." The ratio is a little different on Google Scholar, where now "Gaussian distribution" has half the hits of "normal distribution" (but only a quarter when "inverse" is excluded). These results suggest (1) "normal" is more popular but (2) "Gaussian" is widely recognized. Looking at the results suggests that phrases like "asymptotically normal" may take a long time to be replaced, if ever.
Don't you mean that the null hypothesis is "there is no wolf"?
Simple counterexample: let A=D and B=C, with A and B independent, whence C and D are independent.  Now E=AC=AB and F=BD=BA are identical: as far from independent as you can get.  This shows that trying to dodge the problem by ruling out degenerate RVs or discrete RVs is not going to work.  As a generalization, consider A,B,C mutually independent and D=A, entailing E=AC and F=AB.  Evidently E and F aren't going to be independent in general.  This shows that five of the six pairs from A,B,C,D can be independent, but the single dependence between A and D can make E and F dependent.
Thank you. This seems to support my intuition. I have found [inverse probability weighting](http://en.wikipedia.org/wiki/Inverse_probability_weighting) on Wikipedia, and it seems that the [Horvitz-Thompson estimator](http://en.wikipedia.org/wiki/Horvitz%E2%80%93Thompson_estimator) is a special case of applying i.p.w. to estimate the mean. I still need to figure out if it matters for my analysis that the household-level weights are now according to a SRSWR strategy.
@whuber: I tried to help without spoiling everything.
...different concepts that are logically distinct. :)
(cont) in which case it's certainly not a random sample from the population at large but it unbiasedly characterizes the population of "people who have been to the hospital".
This is just the information I needed, thanks! The categories in both sets are unordered. Sorry about the confusion re Pearson's residuals being independent of the chi-square test - what I meant was is it ok to just report the Pearson's residuals without the chi-square details (no need for an answer to this now as I think you other responses have covered this). Thanks again!
Thank you...to the point.
Are you just looking for R code?
I should be more precise, since I might be mis-using the word univariate. I mean would just running t-tests and chi sq/Fisher exact tests individually comparing the variables between cohorts be sufficient to present the results, or would doing cond logisitic regression add anything? I think that is what you're answering, and what I meant to ask, so thanks.
it looks this problem has been widely known since 1939, called Jeffrey-Lindley paradox. Gelman's book (p185 & 250) also says uninformative priors w/ Bayes Factors are a no-no. Basically, they say that formulating point null hypotheses are stupid, and tests should be constructed from the posteriors. http://www.artsci.uc.edu/collegedepts/economics/research/docs/Wppdf/2009-01.pdf
No, that is a modeling choice.  It is certainly more general to include the additional interaction term (if there is no additional effect, the estimated coefficient would be close to 0), but I would say that it is more common for researchers to leave out  interaction terms unless they specifically have a requirement for it.
But, I thought that when it exceeds the critical value (4.41 > 2.365), hypotesis is definitely rejected!~
I really don't understand what you are trying to do. Do the lists represent input vectors or labels or ...?
@SincoleBrans Please see http://en.wikipedia.org/wiki/Mean_squared_error, section "Regression".
You could always bootstrap, using the `boot` package.
Any answer or comment that claims the mean and sd of a dataset are unrelated is plainly incorrect, because both are functions of the same data and both will change whenever a single one of the data values is changed.  This remark does bear some echoes of a similar sounding statement that is true (but not terribly relevant to the current question); namely, that the sample mean and sample sd of data drawn independently from a normal distribution are *independent* (in the probabilistic sense).
"*a number of the tests for normality that I could use*" - off the top of my head I can only think of one suitable "off the shelf" test (perhaps arguably a second), though I can think of a bunch that would work if you specified, rather than estimated, the parameters. Which ones were you thinking of?
,yes sir,i agree with you.But the problem is after building model using OLS we have only one equation of y=beta*X+intercept. But with quantile regression,(if i take 5 quantiles into consideration) we have 5 such equantions y=beta*x+intercept corresponding to 5 different quantile values.So which equation i have to consider for future predictions..
Thanks @probabilityislogic. I have included a plot in my report, and it sounds like that plus my original sentence should be sufficient in this case. I think that including diagnostic plots is probably unnecessary for this audience, though I've obviously checked them myself and they look reasonable. Thanks also for the explanation of R2 vs. RMSE - it's very helpful.
So for example, you use the data to get to *t* which is related to a distribution, which gives you *p*, which in turn yields a binary inference about a population parameter. So from a frequentist perspective, t, p, and the binary inference are all random variables. All were involved in the inferential process. I'm not sure what the pros and cons are of labelling all or only some such statistics as inferential.
I should perhaps mention that the title is literally a duplication of http://stats.stackexchange.com/questions/33888/x-and-y-are-not-correlated-but-x-is-significant-predictor-of-y-in-multiple-regr
I agree that whomever left the downvote(s) should leave a remark as to why this was downvoted. To Irishstat, it would be much easier to read your post if you took advantage of the formatting options for leaving answers, especially those available for marking up equations in latex. See the [markdown editing help section](http://stats.stackexchange.com/editing-help). That link is available whenever you type a response in the top right corner of the posting box (in the orange circle with the question mark).
@JasonMorgan i suspect that with datapoint, he refers to a single time they measured the sample. Not to the number of observations within a single measuring session.
@gung I think the big problem with the Mann-Whitney on this tiny a sample size (3,3) is that - no matter how vastly separated the samples are, and even without any ties - you simply can't get a p-value below 0.1 (two tailed). At least a suitable parametric test (if its assumptions could be argued to be reasonable) has the *possibility* of reaching an interesting p-value. One needs to bring some assumptions (not necessarily normality) to this small a sample, and try to justify them.
It is not quite clear what your questions is. Is it, "does it make sense to use VIF in assessing formative indicators?" (to which the answer is probably no, or with reservations -- it seems an odd measure to me, and it was designed for a different problem in mind), or "What is the button that I need to push in SPSS?" (in which case your kinda longish introduction is not that necessary). I would suggest taking a look at Bollen & Davis (SEM, 2009) and Bollen (2011 MISQ) for a different take. Given a less than perfect English of this student paper, I would have reservations advising to follow it.
+1, clear and comprehensive, a better answer than what I gave.
If you use a probit, a multivariate extension is straightforward as a multivariate probit can be estimated. Dependencies would be implicit by the covariance matrix of the implied multivariate normal distribution.
@DimitriyV.Masterov There is no real random price variation, but of course product prices changed over time. The price changes don't occur at one point at a time, so it might be that some prices are stable for 2 weeks and other prices were stable over a month (also a difference to the panel data).
The question of particle filtering arbitrary graphs is really interesting. I once found an excellent thesis which describes some solutions: [here](http://www.cs.ubc.ca/~murphyk/Thesis/thesis.pdf)
According to this thread: http://www.mail-archive.com/r-help@r-project.org/msg116971.html , `offset = log(population)` should be the equivalent of Stata's `exp(population)`.
I want to clarify my problem. My dataset contains human likes or dislikes of some items. Thus my table contains 1, 0 or NA. I want to impute all NA values. Initially just a small amount of cells are filled but with time we will have much more values (instead of current NA). We want to constantly fill missing values with imputed. Maybe there might be 1 imputing algorithm for situation with just few filled cells and another for many filled values?! Did anyone have such experience of dynamical imputing procedure selection?
Thanks.  I've done some more searching and your formula appears to be more commonly used (and using a ratio of variances seems simpler and more intuitive to me).  I tried both formulas on two data sets; the results were qualitatively similar in one case and quantitatively identical in the other, so I guess the results of the two versions of the test converge at some point.  I'm going to proceed using your formula.  If anyone else has any insight into the two formulae, it would be great to hear it.
"V-shaped" is not a precise term. There is no way for us to know whether you mean something that looks like $f(x)=|x|$ on $x \\in [-1,1]$, or $f(x) = 1-|x|$ on $x \\in [-1,1]$, for instance.
I don't agree that "null event" is typically called the impossible event. The impossible event is the empty set and it can _never_ occur. A null event is an event that is assigned the probability $0$ by the probability measure; it is _not_ necessarily the empty event. For a _continuous_ random variable $X$, the event $\\{X = a\\}$ is a _null event_ because it has probability $0$. But the model does not deny the possibility that $X$ will take on the value $a$. $X$ _might_ take on value $a$; just not frequently. The relative frequency on $N$ trials is $o(N)$ and converges to $0$ as $N\\to\\infty$
are you shure you are permitted to distribute the paper via dropbox. better double check.
That test does not have a $\\chi^2$ distribution but rather a degenerate distribution because the degrees of freedom increases with $n$.  To better understand accuracy, look at measures of calibration and predictive accuracy.  See http://stats.stackexchange.com/questions/64788/interpreting-a-logistic-regression-model-with-multiple-predictors for a way to obtain multiple interesting estimates.
The short answer is that you basically don't need to worry. The weighting gets taken care of automatically.  If there are a very small number of patients with multiple consultations then you might have a numerical problem, but otherwise everything should be fine.
Thanks a lot, that is what I am looking for.
OK, but in the presence of multicollinearity, the maximisation routine can have problems to converge, or may even fail to converge. These problems can translate into dicely parameter estimates.
@Christopher, if you want to even have a notion of "most powerful" you need to specify a class of alternatives. Is it the class of all distributions $F$ on the real line? All of those with a density $f$ with respect to Lebesgue measure? All elliptical distributions? Etc. The Anderson--Darling (AD) test is, of course, more powerful in general than the KS test because AD can be viewed as weighting the tail observations much higher. So, if you get any unexpected deviations from normality in the tails, the AD is going to pick that up immediately.
Went with stratified approach, boss liked the idea.
This question is loaded with typographical errors.  (For example: the first two equations are identical but shouldn't be; the expressions for the probabilities of the vectors $x$ and $y$ are incorrect; the pdfs are not normalized to unity; the indicator in the integral makes no sense.)  Since you're being quite technical in expression, it would behoove you to check over the text and fix the mistakes so that you are not misunderstood.  Perhaps, though, it would be easier and clearer to express the problem in plainer English and provide some motivation?
+1. Tukey's [median polish](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/medpolish.html) works well for fitting the model when that's all that's needed.  It would be interesting (but perhaps a little computationally intensive) to couple that with a permutation test.
Precision is the inverse of variance. Just a different name. As you've seen on Wikipedia page, it is easier to work with precisions, as they are additive as far as computing the posteriors are concerned. You could check your calculations by having a regression with a single explanatory variable, an intercept. Then your formulae should reduce to the ones on Wikipedia page.
This is essentially the same question as http://stats.stackexchange.com/questions/8285/normalizing-data-between-0-and-1 except your lower bound is -1, not 0.  So: take any of the answers there, multiply by 2, and subtract 1.  But first read the cautionary remarks!
Sue, I'd like to test out your dataset. Can you post it here?
Regarding the validity of a typical visitor and variance, my concern is that the context that often gets assumed about something like a typical visitor.  In my example if the actual times for visitors is very heavy tailed and left skewed than random visitors times will be less than "typical visitors" time, possibly leading to the conclusion recent website changes caused a drop in time on site per visitor.  When in reality the "typical visitor's" time is inflated due to the dynamics of the underlying distribution.
I'm struggling to understand your question. This is how I understand it: A student answers a series of questions you created. He/she can take multiple attempts to get the answer correct. Then, you wish to take the data and determine the student's skill level from 1) how many times they attempted it and 2) how difficult the question was. Is this correct? Will the student have infinite attempts so they always end up getting the correct answer eventually?
Note that you can also adopt a strategy of sampling from $p(x)$ and then from $p(y|x)$.
Have you considered an ARIMA model? Stats might have some nice features to tune the parameters interactively until you have some acceptable model. Given your example you will have to play with all the 3 components, AR, MA and detrending.
A normal distribution does not have lower or upper bounds.  The simplest solution would be to have a PDF made up of four uniform distributions but you may think it does not illustrate the point.
"Tractable" is a function of your hardware as much as it's a function of your data. Will you be running this on conventional hardware? A supercomputer? A toaster? Pen and paper?
I believe that Trend is what's left over after removing seasonal and cyclical effects and smoothing out irregularity, so it doesn't quite fall into the frequency hierarchy of Cycles < Seasons< Irregularity, which makes it a bit tricky in my mind. I've also seen a combined TC (Trend-Cycle) factor used, where TC < S < I. (I.e. I is highest frequency, S is lower frequency, and TC is lowest frequency.)
Memorylessness says $\\Pr(T>t+s\\mid T>t) = \\Pr(T>s)$.  That's the same as $\\Pr(T>t+s\\text{ and }T>t) = \\Pr(T>s)$.  The event $[T>t+s\\text{ and }T>t]$ is the same as the event $T>t+s$.  Hence the conditional probability is $\\Pr(T>t+s)/\\Pr(T>t)$.  Memorylessness says this is the same as $\\Pr(T>s)$.  Hence we have $\\Pr(T>t+s)=\\Pr(T>t)\\Pr(T>s)$.  A monotone function $g$ that satisfies $g(t+s)=g(t)g(s)$ is an exponential function.  And monotocity follows from the fact that $\\Pr(T>t+s)$ must be less than $\\Pr(T>t)$ because the former event implies, but is not implied by, the latter.
Thanks for your suggestion but ideally I am looking for published material with a real world application... but this seems difficult to find.
@cardinal: to me it should be an answer, namely that it is not the purpose of the forum to do someone's homework. Comments are prone to be overlooked.
+1, this is a very robust, universal approach.  I have often used it.  I'd like to suggest another feature: You can generate data for the max $N$ you would consider, then fit the model for proportions of those data by successively fitting the 1st n of them at regular intervals up to $N$ (eg, n=100, 120, 140, 160, 180, & 200).  Instead of saving a p-value from each generated dataset, you can save a *row* of p-values.  Averaging over each column gives you a quick and dirty sense of how power is changing w/ $N$, and helps you hone in on an appropriate value quickly.
Good point. From the definitions of ARCH(1) process (which you neglected to state) we have that $\\sigma_t\\sim F_{t-1}$ and $\\varepsilon_t$ is independent from $F_{t-1}$ so we can surmise that $\\sigma_t$ and $\\varepsilon_t$ are independent. However the tower property is used extensively when dealing with ARCH processes, so the reason might be purely pedagogical one. Also all the time series textbooks I've seen prove this property using tower properties, so another reason might be that this is tradition.
For `rms` do `?predict.lrm` to see how to get all probabilities of interest.
Right, so this answer concludes that the probability exceeds $4((11-8)(12-11)-2^2)/(12-8)^2$ = $-1/4$!
Maybe a matter of preference, but to be mathematically unambiguous, I would say that your conditions are those for *weak* stationarity
Sorry for not being clear in my previous problem statement: I changed the notation for data from x to z...
 
That's it. Should be very clear now... Thanks!
 
Data: z1, z2, ..., z_{n+1}
 
y1 = z_1,z_2,.........  z_n 
y2 = z_2, z_3,......... z_{n+1}
 
x1 = 1, ..., n
x2 = 1, ..., n
 
y1 = A1+ x1 * B1 + epsilon_1
y2 = A2 + x2 * B2 + epsilon_2

H0: B1 and B2 are statistically significally different...