tttjjj/elt_bench huggingface dataset:
Hello Wayne, right, it is about applying the VAR to difference-stationary data. One estimates a VAR for difference-stationary data, and then checks for possible cointegration applying some tests to the residuals of the estimated VAR. And then, if they are fulfilled, continues the procedure: but I don't understand why not just stop here and use the estimated, valid VAR?
Good response. Although I'm confused on one aspect. I thought technology bad preference shocks could be in many cases 'structural' rather than transitory. So you're saying that these, permanent shocks, would not be captured in the error term? Will they be a variable then?
Brilliant answer. I'm wondering if its possible to deal with variables that might have some known correlation? I'm assuming that you would randomly generate the first variable $A$ then generate the second variable $B$ assuming the correlation?
@omidi your reasoning is flawless.
auto.arima is in the forecast package, isn't it?
... ctd For example, are you trying to estimate an average? You may need an estimate of the standard deviation for that, in which case you might need to have a pilot sample in order to estimate the sample size you need.
The expectation is not the same as a realization; it doesn't imply that the test is 'trivial' in the sense you indicate.
@f_vazpinto I simulated some data with the structure you describe and I've added more specifics to my answer.
@Srikant-vadali: Has my updated answer helped?
Yes regarding determining the prediction formula.  I am not sure where you get the 12% from.  Several of the variables in the fit table are informative about how good the fit is .  Also the R square of 0.286 means that approximately 29% of the variation in the response is explained by the model.  That isn't great but it is still significant.  All the coefficients in your model are statistically significant with p-values less than 0.001 except for one which is at 0.039.
1) From the text it sounds like you want to fit the Date:Year interaction, but not Site:Year or Site:Date; however, the `lm` code has the Site:Date interaction but not the others.  2) Why do you think there's no Site:Year interaction if you've only fit two separate ANCOVAs?
These are questions to be considered, read about, and discussed *for the long haul*. The answers depend on situation, personal preference, research purpose, audience, etc. Woe unto those who seek shortcuts to all learning on a Q&A site!
Nice graph, what software/package did you use to create it?
This is the temporal version of the [Totalitarian principle](http://en.wikipedia.org/wiki/Totalitarian_principle).
If you want to estimate the dispersion parameter $\\phi$, where $\\phi\\ne 1$ then u need to use the quasi-likelihood approach. For example for the poisson glm, u need something like: `glm(y ~ x, family =quasipoisson(link = "log"))`. When the $\\phi=1$, then the quasi-likelihood will be the ML estimate.
If you observe that a word can occur multiple times in a song, which implies a word count divided by a song count can exceed unity--impossible for a probability--your questions answer themselves.
Thanks for your nice explanation :)
I think I cannot use this similarity to reach my goal. I should think about that.
good solution but it again demonstrates that you can't describe this with a single number.
Yes Dilip, that's right.  Sorry about the discrepancy with the question; I should have checked first and I should have defined $\\sigma_x$ (which I intended to do but forgot).
@amoeba You are correct and I apologize for posting a comment that was so misleading (since deleted). Please see my remarks after the answer posted by cbeleites, which apparently crossed yours in the ether.
+1 from me to Weka.
It's not particularly clear from your discussion what you're really aiming to capture there. The difficulty is one could sit here making a hundred guesses about what you want; I think you need to be able to articulate its properties more completely. What underlying problem does this solve?
I have actually used EEGLAB in the past, although mainly it's underlying functions rather than the full GUI (which tended to be a bit slow and unstable). However it's mostly geared towards mass univariate analysis rather than multivariate analysis, although I admit I haven't looked at it in a while. Do you have experience of it?
This should be posted to the statistics SE site.
As always, nice answer +1. However, since we have no information about the sample size, the asymptotic distribution might not hold. If there is no way to obtain the exact distribution though, I suppose I'll have to make do. Thank you very much.
In reference to @andrea answer, how do you know one violates the assumption and the other does not?
@whuber [WolframMathworld](http://mathworld.wolfram.com/StandardDeviationDistribution.html) seems to describe the answer. The distribution of the sample standard deviation is indeed closely related to the chi-distribution.
Also yes, tract = structure, and I have 30 patients and 30 controls.
@post-hoc, it depends on your audience and the scaling/units of your variables.  I agree that you should not use an arbitrary cut-off, but something that makes some objective sense and is chosen before hand.  For some audiences the difference between the averages of 2 groups may be more easily understood than the slope, especially if a 1 unit increase in x is not a representative measure.
I'm not sure what you are talking about... Lacking the context to ask sensible questions is not intrinsically related to you having knowledge of research or statistics. Are you asking whether statisticians can (or have in the past) led projects in health-related field?
I think you're talking about relative risk. That should help with searching.
Binary variables are neither necessary nor sufficient to suggest ridge regression.
+1, there's good information here. I would be cautious regarding the statement "Predictive modeling is what happens in most analyses", however. Whether or not predictive modeling is more common will vary by discipline, etc. My guess would be that most modeling in academia is explanatory, & that a lot of modeling / data mining that is done in the private sector (eg identify potential repeat customers) is predictive. I could easily be wrong, but it will be tough to say, a priori, which happens most of the time.
Ah, what a disappointment...! But this makes the lengths a random variable, which opens up interesting avenues...
if the yaxis is c(0, 0.05, 0.10, 0.15) and xaxis c(5,10,15,20) and mean being 12.5. Would you explain this charts as there is 15% chance that mean would be 12.5? I am having a difficult time understanding the y-axis?
+1 for pointing out Hadley's wiki.
Anton meant to say, I think, that he is given that for the sample of soze n all outcomes are failures.
The beta distribution will never generate a value of 1, as it is a continuous distribution and therefore the probability of generating any prespecified value (such as 1) is 0.  Are you sure you need to be able to generate an estimate of 1?  Or is your real objective to be able to generate a random "sample" of 400 1s based on an estimate derived from the real sample of 400 1s?
Fun question, btw =)
I added some further discussion to my answer. By the way, I love your question, it's a wonderful brain-twister.
I already tried that, but I'm stuck at how to do a double integration in R, not on paper
It is certainly *possible* to do this, but it is hard to imagine a situation where it makes sense to do so.
My only quibble on the ML/DM distinction would be that I think DM is broader.  For example, OLAP and related tools include mining technologies.  But these come from the database side of computer science rather than machine learning.  The role of commerce in shaping the "meaning" of data mining is hard to ignore -- it brings in elements of management sciences, operations research, machine learning and statistics as required.  It also gives the impression of something flimsy, but that's usually a problem for purists not practitioners.
For those interested in confirming my results, per the approximation formula, my derived variance of the sample standard deviation is: (1/(2(n-1)) + kurtosis/4n)*sigma^2.
I think GEE is based on maximizing a quasi likelihood function. You can find details in Liang and Zeger (1986), Zeger and Liang (1986) and Prentice (1988). As explained in Fitzmaurice et al. (2004), “the term marginal in this context indicates that the model for the mean response depends only on the covariates of interest, and not on any random effects or previous responses.”
Could you provide a working example with R code, as requested by the OP?
What's the integral of $0^2$ between any pair of finite limits?
Please explain your need for residuals.  These are not in intrinsic part of the model.  In terms of how to state predictions, providing the predicted probability that $Y\\geq j$ for various $j$ is a good way, and requires no arbitrary decisions.  Looking at the histogram of predicted probabilities you can judge how useful the model is, i.e., how often it provides more definitive probabilities that are closer to 0 or 1 than to 0.5.
Sounds as if you should be asking the person who gave the assignment.
@shabbychef: Ditto.  I think I got lucky this time in guessing the intent.  We have to recognize that non-statisticians often are unaware of the technical distinctions we make.  After all, a great deal of statistical consulting amounts to figuring out what someone might *really* be asking you!  Here the best clue is that the OP himself suggested a nonlinear re-expression ("link").  The scatterplot he posted helps immensely in understanding the situation, too.  The combination of those useful clues seemed sufficient to venture an answer rather than to keep probing for clarifications.
The only slight problem with this approach is that there are no chiptype specific annotation packages for Agilent drosophila chips. But it can be overcome, as the Agilent output chip itself maps the probes to genomic positions. Or I can do it the hard way and go through biomaRt and annotate the genes (not probes) to cytobands on my own.
Welcome to the site, @Remi. This isn't really an answer to the OP's question. It is more of a comment. Please only use the "Your Answer" field to provide answers. I recognize it's frustrating, but you will be able to comment anywhere when your reputation >50. Alternatively, you could try to expand it to make it more of an answer. Since you are new here, you may want to read our [tour page](http://stats.stackexchange.com/about), which contains information for new users.
@ndoogan: I couldn't find information on this
@whuber Thank you for getting me started. I'm still new to stats so I had to scour wikipedia and my long forgotten calculus for help.
Can you define "similarity"? In your example, would you consider the points "similar"?
Just curious: How much data do you have at your disposal and what kind of software are you using for your analysis?
The example calculations make almost no sense: Your row totals are correct, but column totals should be 1000 and 2000. Thereafter expected = row total $\\times$ column total/table total, so the first expected frequency should be 300 $\\times$ 1000/3000 = 100, and so forth. A tell-tale sign of misunderstanding is the observed and expected frequencies don't have even roughly the same sum, as **all** the expected frequencies are less than the observed.
It is difficult to discern what you are asking.  Could you perhaps give an example of what the answer would be with, say, three names?
@Glen_b Sorry but I only have a basic knowledge of stats, do you have some hints on how I can implement a survival model on top of a markov model?
could you please explain "non-linear interactions". The Spearman Rho it seems reflects a measure of validity coefficient in terms of psychometry. I do not know about the nature of Tau.
I would guess that the hospital visits per se may not matter that much; rather, whether an antibiotic was prescribed during this visit, and/or whether it was a new antibiotic. Having compliance data would have been wonderful, but I doubt it can be obtained. If you have prescription data, may be you can compute cumulative doze: `bysort patient (time): generate cumdose = sum(dose_prescribed)`
This interesting answer, which seems more like a comment to the question, might be more appropriate at [Is R2 useful or dangerous?](http://stats.stackexchange.com/questions/13314).
differentiating to find its maximum? that is when all probabilities are $\\frac{1}{L}$
It is unclear to me what the actual problem is that you are attempting to solve by collecting this parallel dataset, and how exactly this parallel dataset will solve that problem for you. Right now I read all of this as saying something like "I think my dataset is not great and I think more data would be better." It's hard to know how to respond to this. Maybe you can explain more clearly, directly, and precisely what you are trying to accomplish.
That paper seems to be getting a lot of press recently.
For a stationary model an OLS regression with seasonal dummies is easier, and has the advantage of being easy to interpret.  But it assumes that the seasonal effect is fixed and uncorrelated with longer time trends.
The suggestion below `method="none"` is the better solution.
Yes, but it can only be used for interpolation. Suppose datapoint $x_{ab}$ is missing and that the data matrix is centered. The problem is to find $\\mathbf{V}$.

You can estimate the sample covariance matrix as:
$$\\hat{\\Sigma}_{ij}=\\alpha_{ij}\\sum_{k}x_{ik} x_{jk},$$
where $$\\alpha_{ij}=\\left(N-1-\\sum_{k}[\\delta_{ab}(i,k)+\\delta_{ab}(j,k)]\\right)^{-1},$$ and $\\delta_{ab}(i,k)$ is one when $i=a$ and $k=b$ and zero otherwise. Given this, the transformation matrix $\\mathbf{V}$ is straightforward to obtain (you just have to estimate the SVD decomposition of this sample covariance matrix).
+1, nicely done Maxim. Much has been previous discussed [previous post](http://stats.stackexchange.com/questions/10/under-what-conditions-should-likert-scales-be-used-as-ordinal-or-interval-data?rq=1) We could add examination scores to routine violation as well.
Any linear regression should have roughly normal residuals, but a lot of data/model combos won't have normal disturbances.  If you're only interested in coefficient estimates and standard errors, consider using a heteroskedasticity-consistent (aka "robust") vcv estimator.
Do you know the functional form of $f$?
@marino89: 10-fold cv might not give you the best values for the parameters. Maybe you could "roll-through" each training period 10 observations at a time, always testing on future data?
I just answered the personal email you sent on this topic.  Please do not ask the same question in two different places.  My answer was:  I've seen worse fits, but it's a tough call.  You always have to ask if another approach would make things better or worse (because of higher variance from more parameters if sample size is not huge, or because of worse fit in other ways).

To relax that assumption you need the R `vgam` package.

I don't use Hosmer-Lemeshow.  There is a better test from Hosmer and le Cessie - see the help file for `residuals.lrm` in `rms`.
Some closely related threads provide several good answers to all your questions: http://stats.stackexchange.com/questions/30728, http://stats.stackexchange.com/questions/1601, http://stats.stackexchange.com/questions/24227, and http://stats.stackexchange.com/questions/41361.  More can be found by searching for [log transform](http://stats.stackexchange.com/search?tab=relevance&q=%20log%20transform). If you wish to follow up, just edit your question to focus on any remaining issues that need to be addressed.
It has in fact gotten cut off, fixed it.
My advice is to stop thinking about statistical significance and start thinking about effect size. In a 2x2 table such as you seem to have, the odds ratio is a good (and easy to calculate) measure of effect size, just find the cross-product. Most software will have tools for confidence intervals around these, if you want them.
To see exactly how the contrasts are specified, try running `contrasts(CategoricalEffect)`. You should see two columns of contrasts coefficients, where A is set to 0 in each column, and therefore acts as the reference level for both comparisons.
@StasK, Could you explain which stuff is not in the above books?
Brett's definition between internal and external validity is accurate, but for our purposes it will help to define it in different terms. External validity is only concerned with the sample and how that sample relates to other populations. Internal validity is concerned with various aspects about the effects estimated and the constructs used to estimate those effects.
+1 QQ-plot is great for comparing the distribution of two variables.
Normally I would agree with you, but in this case the specifics might matter since there could already be a bioconductor package that does what you ask.
Sure.  http://www.rasch.org/rmt/contents.htm http://www.winsteps.com/minifac.htm  Setting up the specification files can be awkward.  Let me know if you want me to run it for you and send you the output.  The data would need to be in a csv file or an Excel file. Good luck with this - it can be very interesting.
@niandra82, the two algorithms have already processed every algorithm in N and I can in fact not run them on any set smaller than N - classification is dependent on the whole input set, not a pure function from an element to a category. What I can do is manually analyse any subset of N and decide which one of the two algorithms (if any) picked the correct category for each item.
A DAG has an ordering. You talk about parents and children. With undirected, you just talk about cliques. Normally you just convert it to the equivalent undirected model by moralizing the graph anyway.
You may find this thread interesting as well: [does-an-unbalanced-sample-matter-when-doing-logistic-regression?](http://stats.stackexchange.com/questions/6067/).
It is not an SVD on XY', it is an SVD on a sum of matrices of the form $x_i*y_i'$. Note the expectation! (reminds a little bit of PCA, where you compute $E[XX']$.)
@Stéphane Laurent Thanks for replying. I had a mistake and I didn't recognize it. $a=\\mu$. I will correct it.
Thanks cbeleites - I added those clarifications because I found the wording of the original question a bit confusing. I hope my edits were for the better - I tried to highlight the dilemma better  - but please let me know otherwise. All that said, when you mentioned that you find the difference bogus - I would like to note that @user13420 gets two substantially different results at the bottom of his OP when following approaches **1** or **2**. I have found myself facing this dilemma myself. I believe the **2nd** approach is more common though, but it would be great to get your take on it.
While I agree that GWAS studies involve many statistical tests, how do you get $10^{13}$? There are around $3\\times10^9$ base pairs in the human genome, around $10^7$ single nucleotide polymorphisms, and the GWAS studies I'm familiar with genotype $10^5$ to $10^6$ of those.
I posted an answer related to the informal (underground) economy.  "Black market" can be used to describe such an economy, or it can imply that the market is for trade which, if reported, would be illegal (as opposed to trade which is only illegal because it is not reported/taxed).  Could you clarify which definition you intended?
Why did you say of repeated measures? I don't see it here. It sounds to me from your description that you a kind of give dose I to mice 1,2,3; dose II to mice 4,5,6, etc.
@swisslog, I only changed the order of some terms in `MAvar`, compared to the last equation, so they should be the same. I have added two more lines to the derivation, the rest is a couple of geometric sums and simplifications. Now regarding N, the thing is that we are looking at a theoretical variance, not empirical one and, since in theory this smoothed AR process never ends, the theoretical variance does not have to be zero as well as it does not depend on N.
Adding a confounding variable ought to make it clear that the original independent variable is not needed. e.g. that you ought to model ice cream sales by temperature, not shark attack.
Thank you for this detailed answer. I ask the same question as above, what if I can't make the assumption that the spatial correlation is independent of the location? Is it correct that modelling the variogram is then not useful, as I would have to make a model of the variogram for all locations? Is it then better to use IDW?
Could you make available your example data set?
I would like to try something for you. Regards,
Eric
I haven't, although, I will try it out some time. There are often other ways to approach multiple class problems using single class classifiers.
Answered there as well.
I'm sorry I can't even read that. You're welcome to type in a short quote if you want your answer to be legible, but it would also be a good idea to add some of your own thoughts, e.g. interpreting the material in the light of the question.
Side note: You can replace the square within the median by any monotone increasing function $g$ of $|x_i-y|$ without changing your optimal solution.
I think you answer your own question. The logic seems reasonable - for practical purposes, if you only need to run a computing task once, then the length of time is fairly irrelevant unless it's so long that it can't even be performed once. You may want to consider the possibility of tweaks to the classification tool, however. How quickly would you need to be able to respond to new topics appearing in the text? Would you ever need to go back and re-classify old text? etc.
What do you mean when you say that you want to test for correlation?  Do you mean that for each of the 200 genes you want to do a separate test that the correlation between the treated and untreated groups is different from 0?  That would mean doing 200 tests and would have the usual problem of multiplicity (which is a serious problem when doing so many tests).  On the other hand you might want to test for a single correaltion between groups ignoring what the specific gene is.
Did you want to add that not all of the variance will be explained by gender, e.g. some could be due to age, or quality of public transport. :)
@mpiktas- For the first example, how can a *non-negative* random variable have a mean of 0?  Only if it is a constant (=0), then the approx. is exact!  and I said *finite variance* case, which rules out the other example.  I did say when the approximation was good.  Find an example which is in the class I specified in the comment above (non-negative, finite variance, low skewness).  And besides, the approximation works well *in the particular case being described*.
+1, this question is very nicely presented.
@Felix sorry I forgot to flag ("@Felix") that response back to you, so I am doing it now. That above response might read more dismissively than I intend -- it was a good question to ask.
@PeterFlom I already knew that SAS gives odd results in such situations, I'm suprised by the R outputs.
(+1) But your last sentence seems to imply the KS test isn't a goodness-of-fit test, which it very much is. And of course it's still an option if you get the distribution of the test-statistic right when parameters are estimated.
I answered my own question though already. So, it's not really necessary and it's a stupid question. Now I am stuck at something else. At the inflexion point x=1, the bias must be zero right? This is because f''(1)=0 and thus the bias disappears?
In general, select the one that you feel answers your question most directly and clearly, and if it's too hard to tell, I'd go with the one with the highest votes. Even it's a bit painful to decide which one, it's important to reward someone who took the time to answer.
In theory yes, but it seems difficult to construct anything more useful (e.g. confidence intervals) without any further assumptions on things.