tttjjj/elt_bench huggingface dataset:
1) "The vectors of y-values do not all follow a normal distribution, therefore I need to implement a non-parametric regression" is a non-sequitur. Linear regression is reasonably rubust to departures from normality with large samples, and as you're in genetics I suspect you have a *very* large sample. And non-parametric methods are not the only alternative; you could apply a transformation, or use generalized linear models.
2) You still haven't given us any idea information about $Z$, e.g. how many levels does it have?
What have you done (other than finding a solutions manual)? Where did you get stuck? Could you please provide more detail? This looks like a cut-and-paste homework assignment with no context.
"the interval either does or does not contain the parameter θ", but we do not know. The fact the event is completed does not matter, since we do not know the outcome.
To me the fact an event is completed make it meanningless to talk of probabilities because the fact it is completed brings a certitude (the probability the event occured was 1 since it occured).
But when we do not have the knowledge, we do not have this certitude so to me this make sense to keep talking of probabilities.
Nice addition.  Why do other fields keep inventing new names (e.g., precision-recall) for old concepts?
I wouldn't bother formally testing normality anyway. It doesn't answer the question that your really need to worry about (which is not 'is my data normal?' - it almost certainly *isn't* normal, you don't need a test to tell you that, it's 'how badly could my inference be impacted by the kind of non-normality I have?', which the test doesn't tell you about).
The edit even more clearly shows the need to transform $\\log(xy)$: not only are the SDs of the residuals proportional to the square of the fit, they are strongly positively skewed.  If $x$ is supposed to depend on $y$, then consider re-expressing $x$ alone and including $y$ among the explanatory variables.  These plots strongly indicate something like $1/\\log(x)$ or $\\log(y)/\\log(x)$ or $1/(\\log(x)+log(y))$ or $\\log(y) + 1/\\log(x)$ are what you should be investigating.
Yes indeed, forgive my phrasing I was just outlining a quick way to get to the desired result. You certainly can get zero covariance without independence.
Ah but of course! How could I have forgotten! Everything works fine now! Thanks for the links as well, the first one seems especially helpful :)
(ctd)... searching on the italicized terms here (and more broadly) should turn up a number of links. It would be worth considering.
Or your husband/father....
@joriki, accounts issue aside, welcome to stats.SE!
Try `help(anova.lm)`.
Also Bayesian approaches to testing can help. What I'm not interested in is a dichotomous decision "either difference or equivalence". Statistics should address situations too bad for reliable inference (e.g. too small sample sizes) as outcome (3) does.
Please merge this account with your [original account](http://stats.stackexchange.com/users/10619/subhash-c-davar). You can learn about how to do this at our [help page](http://stats.stackexchange.com/help/merging-accounts). Please **stop** creating a new account every time you vist the site.
I don't know... Paul provided xy coordinates but not SD for all the points. We should see a graph with 5 points with perhaps some error bars (assuming we know the SD).
It sounds like the only issue here is the choice of how one structures such data. Look at time, for instance, if one is interested in describing an AR-1 process, you might merge a time-series with it's lagged effect. For AR-2, merge two lagged effects. For universal kriging, merge a point, it's lagged effect, and adjacent samples: just one way of representing correlations in 4 dimensional data. Covariance is, by definition, only computed between two random variables.
(+1) That's a nice solution. Should you have $t \\geq \\cdots$ instead of $t \\leq \\cdots$ in the display equation?
Could you please explain what $g$ and the $S_i$ are?  It's best not to make your readers guess.
@Matthew, you lost me. I can't see how one could fit a finite mixture of normals to something like this.
Brilliant, thanks for the info. Indeed I meant covariance stationarity.
The mediation effect is really the difference between 1 and 4. That is, how much does adding the mediator change the parameter estimate in 1.? Rather than say "it does" or "it does not" the new view is to say "it does it this much". MacKinnon has a website that has a lot of material and some step by step instructions. A lot of this is referred to in other answers you've already gotten
i don't understand what you say. If you want to compare group means, the procedure is correct.
You're using the term "death" and "censoring" (for cancellations and closures) interchangeably here when they're not. You need to clarify: when someone experiences a "death", they are no longer at risk for a "death" event, however when someone is "censored", they continue to be at risk for "death" even though you wouldn't observe it if it happened. Can you revise the question, skipping statistical jargon, and try to explain the problem?
@Leo: Note that I use `<x1, x>` and `wTx`. The former is used as a symbol for a kernel evaluation K(x1, x), which projects x1 and x into a very high-dimensional space and computes implicitly the scalar product of the projected values. The latter is the normal scalar product, so `w` and `x` have to be projected explicitly, and then the scalar product is calculated explicitly. Depending on the choice of the kernel, a single explicit calculation may take much more computation than many kernel evaluations.
Thank you very much for the help - this is one of the issues when textbooks make errors, as when learning it can be impossible to verify!
2nd Challenge: The expert has given a wide range for the parameter X, say, 10% to 90%, and my data (even after I have a fair enough sample for that partcular parameter estimation) gives a value of 12% with a range of 5% to 25%. No way I see this this parameter X go upto 90%.
Thank you for helping me clarify this a bit further. I have updated my answer, although I am still not clear. My question is 'what is the best estimate of $\\mu$ given the prior and data?'; there are no other parameters.
@AdamO: In model selection usually some form of regularization such as parsimony is employed, this provides uniqueness.  If two models really are equivalent, then you can define an equivalence class, but in application there is usually some form of regularization such as distribution preference.  I also don't understand you're interpretation of splines.  Densities are functions, and splines in particular low rank splines are perfectly good at estimating functions, furthermore with low rank splines you can just choose a fixed number of knots and use the coefficients as a family of densities.
The figure on p. 2499 of the above paper is exceptionally useful.  I do have a small issue with the use of data splitting for validation with those sample sizes.  One of the most important things to know is that standard tests of partial association (e.g., likelihood ratio $\\chi^2$ test) are the most powerful for testing whether a new marker adds information.
It's just a starting point, but perhaps check out some of the answers to this question: http://stats.stackexchange.com/questions/2777/modelling-longitudinal-data-where-the-effect-of-time-varies-in-functional-form-be
Another quick question. When reporting results which output should I refer to, the 'Greenhouse-Geisser' or the 'Sphericity Assumed'?
My question is about the maximum and minimum of $X(t)$ between times 0 and 1 in continuous time, conditional on $X(t) = c$.
The square root of the variance is in fact the standard deviation. You need to know the mean to calculate either one.
Agreed. I think statistics textbooks probably present them in that order because the ANOVA is simpler, and the ANCOVA can be seen as an extension of it. If you believe that your additional predictors will explain substantial variance in your model, there's no reason to tie your hands by insisting on finding significance without them. The whole point of covariates is to increase your power to find effects. (unless you're interested in the covariates as predictors in their own right, in which case it makes even less sense to leave them out.)
(Cont.) Eigenvalues, standardized coefficients, structure correlations, discriminant scores - everything will be the same. Only eigenvectors will differ. The reason why there's no effect of standardization on the main results in LDA is that LDA decomposes **ratio** of Between-to-Within covariances, and not the covariance itself having its **magnitude** (as PCA does).
Can you say more on what you mean by $n \\beta$ "put together"? Do you mean finding the sum of their products, as you would to find a fitted value in standard linear regression? To your second question, you'd like the vector to contain the standard error of the $\\beta$ coefficients, is that correct?
Thanks Lockedoff, especially for helping me sort out my own thinking w.r.t the difference between the conditional and joint independence model
You put a lot of domain knowledge into this though. What if you do not recognize the pattern of the time series yourself? How do you then construct a model that can, especially if it depends on inputs that are infinitely far back in the past?
Your latest edit does not rescue anything, because in the counterexample for $c \\gt 1/4$ the matrix is clearly invertible (its determinant is $c-4c^2\\ne 0$).  I would like to suggest you let this rest, instead of applying a series of quick edits; rethink your question; and pose a new one when you have better determined what it is you need to show.
This is no criticism of your efforts, John, but it is sad to realize that you might be successful, because--by not accounting at all for word sequence, sentence structure, or underlying ideas--achieving good predictability based merely on words used would indicate how little the human grading really has to do with quality of writing or thought.
I'm not seeing about a dozen of the 'dials' for random forests here.  Cross Validation?  Bayesian Priors?  Nature of the resampling?  Training set for each tree?  What percent of the subset to each tree?  ... there are lots more that could be enumerated, but my point is that you have other inputs to consider.
+1, FWIW, [?scatterplot.matrix](http://rss.acs.unt.edu/Rdoc/library/car/html/scatterplot.matrix.html) in the [car package](http://cran.r-project.org/web/packages/car/index.html) will give you a similar plot (w/ some differences, eg, w/o the r's & stars).
+1 nice and clear answer. Jeromy, can I ask a question about point 3? I do understand the reasoning behind transforming the data, but something always bothered me about doing that. What is the validity of reporting the results of the t-test on the transformed data to the untransformed data (where you're not "allowed" to do a t-test)? In other words, if two groups are different when data is, for instance, log transformed, on what bases can you say the raw data is different too? Bare in mind, I'm not a statistician, so maybe I just said something absolutely stupid :)
Closely related: [What is Meant by a Random Variable?](http://stats.stackexchange.com/questions/50/what-is-meant-by-a-random-variable).
I didn't,just I read in this book: time series model by Harvey, page 12.
My gut feeling is to just compare the "best" models: the best X1 with the best X2. After all, you aren't going to use the full model if a reduced one is better, will you?
@ phils: This is self-study question (see the tags), you are not supposed to solve them, give some hints next time.
Copulas can have as many dimensions as you like, the examples are bivariate because it's easy to visualize.
Not that this addresses your question, but What is your unit of analysis?  Donor country in a particular year?  In which case what sort of country has zero articles relevant to aid?  Alternatively, is the response the amount of aid a recipient country receives from a particular donor and you are interested in whether media impacts on how the donor govt distributes its aid?  In either case an additional problem to the one your question is about is that there will be two-way causality - media impacts on aid decisions, but aid also causes media coverage.
Also can be downloaded at Tibshirani's homepage http://www-stat.stanford.edu/~tibs/ElemStatLearn/
You can estimate the empirical distribution of X, the empirical distribution of X|Y, and compare the distributions. If these distributions are equal, then X and Y are independent. It is easier if Y is discrete, if Y is continuous, I would probably first discretize it and then estimate the empirical distribution.
@Navi: by definition, the most probably path is a horizontal line around zero. These random walks are just cumulative sums of Gaussian noise with mean 0 and unit variance
Thankfully I've got the sample sizes
As you know, your link is a discussion of *controversies* and mentions most of those positive studies only to point out why they might be of limited use or credibility. Regardless, how do you answer the question in front of us? "How can I attack this problem [of] stochastically modeling" near-term market moves?
The math people asked good questions.  This question's logic seems circular: it appears to ask us for a formula to measure the "quality" of an SE question but it doesn't stipulate what "quality" means except to give non-operational synonyms like "value as perceived by the users of the site."  You can't get something for nothing!
Thank you for explaining that the outlier score is relative to the cluster centroids. I had impression that this was relative to other compounds. Now this restriction makes sense. I'll try to experiment with your other suggestions.
Thank You @Marc
Some information are missing, as suggested already. Also, anticipating dropouts and defining the right hypothesis will change the standard formulae. For an overview, check e.g., Donner, [Approaches to sample size estimation in the design of clinical trials–A review](http://bit.ly/q0clPS), Stat. Med. (1984) 3:199.
Or, considering there are only 0's, 1's ....4's, give us a table.
This seems like an insightful approach, but working solely with *covariances* is not directly going to solve a question about *correlations.*  The problem is that the denominators in the correlation calculation are standard deviations of the variables and those will change, too.
...If you want to do what you are setting out to though I think you need more data, including multiple days worth of data, and response by sub-audience.
I find the mathematical notation employed in the cited "paper" (i.e. presentation)  non-comprehensible without  comments. In particular, what does the line of dashes symbolize? Implication?
@Stask, I'm afraid I don't follow. Wouldn't the null hypothesis be "this model is cannot predict", i.e., rho=0, and the alternative hypothesis "this model captures some info about temperature" i.e., rho>0?
Have you seen [this thread](http://stats.stackexchange.com/questions/25232/ancova-or-repeated-measures-anova)? Can you elaborate on what is the difference to your case or why you are not satisfied with the answers given there?
Good tip! I just looked up the topic, apparently found the original article, and it's referenced 11 times 2010-2012, in a couple of different fields. That's reasonable, though it might also indicate people throwing an interesting technique at a problem in their field to get a publication. As you say, experience does matter.
@KasperChristensen If you think my response doesn't answer your original question, you should probably clarify where it's inadequate. (If you feel instead that it doesn't respond to additional questions in comments, you should either modify your question or ask a new one.)
I was kind of aware of this but... I'm not sure how to measure the reliability of the classification. I can trying agasint the already classified data to see how well it goes but ... on new data? how can I know it will work well, in which cases?
For me, scalar implies contrast with vector, matrix or tensor, which is quite separate from your sense here. Do you have references for this terminology in statistical literature? Also, the definition that categorical excludes ordinal is certainly disputable (contradicted by Agresti's _Categorical data analysis_ and several other texts).
+1, but I'd recommend not leave-one-out but repeated/iterated k-fold cross validation, which allows more than 12 combinations of users into models, and thus a better measure of model stability.
@SvenHohenstein, from his last sentence, I gather that he isn't suggesting that the AIC *itself*, is a measure of model complexity. Brause42, note that the question specifically asks about models w/ the same number of parameters. Thus, the AIC will reduce to SSE or deviance, or whatever.
See answer below. Thank you!
OK, the closest statistic I could find is called the [Location Quotient](http://biomedcentral.com/1471-2288/5/32). However they don't describe a pdf.
I cannot help with your convergence problem, especially not without seeing the data. However, if you need to consider autocorrelation you can use `lme` from package `nlme`. It provides an argument `correlation`, which can take, e.g., an ARMA correlation structure.
Is this homework or such? Add the self-study tag and your thoughts about the problem. Also, you should explain what the 'two equations' are - are they defining $x(t),y(t)$ as stochastic processes?
Thank you very much Greg.
Nice question!

I am not aware of any. One could think of a distance of the generated data and the real data. For example, the Kullback-Leibler distance between the two probability densities. Another possibility could be: if you present both data to a canonical classifier, it cannot tell one from one another (or at least with high confidence).
Hey, Brian! Thanks for your response and apologies for the format. As I mentioned to Joran above, I attempted to use droplevels, but it produced another error (see above). Any suggestions for this one? Thanks!
thanks, I'll think about your answer suncoolsu
This is a [mixture modeling](http://en.wikipedia.org/wiki/Mixture_model) problem.  Naming it is half the battle - I'll let somebody else take care of the other half.
This question has been asked about 10 times already on stackoverflow and this site. Please use the search function.
Can you give more details on the actual problem? What prevents you from doing the same you did in 1D also in 2D?
Are you asking which test you will use when you have the data, or how to determine the sample size you will require to insure a specified level of power given a specified effect size? Note that we don't usually say we need a test to determine power (although, it is true that power will depend on which test we intend to use).
since you waded in robust standard errors, you should mention not only heteroskedasticity-consistent errors, but also autocorellation, etc. In R various variants of such standard errors are provided by function `vcovHAC` from the package **sandwich**. It also worth mentioning that all these standard errors are only asymptotically consistent, which means that in small samples they should be used with care.
thank you Subhash, actually I used four IV, but the other three give a partially mediation using Baron and Kenny method. However, optimism give a no mediation and with a negative path coefficient.
@Gung, Thank you for that link! Rly explains it very well. But I was wondering if there was an easy way to difference the data sets? Or is there a specific method to do this? If so, could you give me the name?
Phenomenal explanation -- many thanks. Can you comment on the generality of this approach? Is this a method often used to compute cross-moments? I asked the same question on Math Stackexchange and got an answer dealing with joint MGFs (i.e. finding appropriate coefficients in the Taylor expansion), which I take it isn't really a *different* approach because it's just encoding the same algebra in a different form. Is this right?
@JeremyCoyle: bias does not mean that you will *necessarily* err on that side. I agree that the risk of overfitting depends very much on the type of problem and data at hand - I come from a field where overfitting is *extremely* common, and we commonly have small sample size situations with serious trouble due to model instability - and would therefore prefer to err towardy less complex models.  I also agree that selecting one complexity parameter is a comparably benign situation: trouble is more likely if more hyperparameters or a more aggressive optimization scheme is used.
Please amend your question, then, to fully reflect what you are trying to do. It sounds like you might be analyzing all of the choices a consumer could make - that is a different problem from the one you posted.
@whuber Now that you put it in those terms, It seems to me that though there may be minor dependence, it should be little enough that I can assume they are independent. I edited the question to reflect this.
An alternative explanation is that your features are derived from the data set (as opposed to being predetermined, as in Lee et al.), and are all guaranteed to be present.  If this is the case, I would think it impossible for you to have negative feature weights.  Generally, the greater in magnitude the weight, the more important the feature is to the classifier.  You should read the Guyon et al. paper in the linked CV post.
What about this alternative proposed ? http://nlp.stanford.edu/IR-book/html/htmledition/multiclass-svms-1.html I am not sure to understand the feature vector they are talking about, is it possible to do more than 2 classes with it ?
This is answered here: http://stats.stackexchange.com/questions/18844/when-and-why-to-take-the-log-of-a-distribution-of-numbers. Also, look at the comments below the question there. One of the commenters provides links for further insight. Also, this might help too http://www.ats.ucla.edu/stat/mult_pkg/faq/general/log_transformed_regression.htm
Your remedy will not help here - centering does not affect correlation between variables.
jonny must be right because in the R code *geno_A* and *geno_B* are interacting each-other to get *y* .
@Tommy Please register your account.
You are, of course, completely right. I was a bit too short there. What I meant to mention was a scenario where a PI who is very determined to prove a point and poor quality data that cannot prove that point (often due to fundamental design issues) get together.
Wouldn't it be preferable to run a paired difference test on group A vs. B (and also a vs. b)? I suggest this simply because it appears that @TheChymera is interested only in one or two pairwise differences with respect to the greater ANOVA (which gives p for *any* group not equal to other groups, which includes differences that may not be of interest).
Best to look at it backwards. If you want to use any method that requires `tsset` or `xtset` first, then you must have no more than one (panel identifier, time) pair. But you can define panels as (country, firm) pairs and that's often a good idea. See http://www.stata-journal.com/sjpdf.html?articlenum=dm0034 for technique. `egen panel = group(country firm), label` would be an example.
Also would this method work correctly if the x & y points on the plot were listed as one x column and one y column in a data frame, or would they need to be placed and arranged into a matrix of dimensions: (highest x or y) by (highest x or y)?
100 is too small for me. I would opt for a leave-one-out strategy for both cross-validation and test evaluation.
When I import the data into SPSS *none* of the variables are integers. All of the ones used here are floating point except for the `FoodInsecure` - which is `F3.1` (in Fortran style number formats). This causes the variables to all have slightly different summary statistics in SPSS compared to here. Here is [SPSS code](https://dl.dropbox.com/s/ru5fii2dzxc5cas/Stata_SPSSDiff.sps?dl=0) to replicate. So it is likely the two programs input the Excel data differently - which is correct I'm unsure.