tttjjj/elt_bench huggingface dataset:
(I hope my imperfect english does not complicate this discussion)
I have also done the t-test (as a posthoc test) - which confirms that there is a difference).
My last idea was to compute the absolute difference from the grand mean of the dependent variable and then subtract the Bottom 2 value from the Top 2 value - which gives me a value between 0 (no difference) and +/- 5 - if it's negative the bottom 2 values are more important and vice versa. This gives rather strong results - most values are around -2 to 2.5. Do you see any fundamental problem with that approach?
@usεr11852 Yea i was trying to figure that part out. Should i even do a comparison between various countries or dive straight into making a point between population and the various variables (energy consumption/CO2/ forest area) ? The video is really awesome and i really like the visualization for the chart with circles moving with respect to the years. Is it possible to do that in matlab ?
Absolutely right. The key is that any measurable function of the regressors can be pulled out of the expectation, since we are conditioning on the regressors. Conditional heteroskedasticity is typically written $\\mathbb{E}(\\boldsymbol{\\varepsilon\\varepsilon}'\\mid \\bf{X})=\\sigma^2\\mathbf{\\Omega}(\\bf{X}‌​)$, that is, the variance matrix of the errors is a function of the regressors. The weights are then just the (square root of the) diagonal elements of this matrix, which means they themselves are functions of the regressors.
(+1) Another good read is Montgomery's *Design and Analysis of Experiments*.
I've read this twice and still cannot picture your situation at all. A small example of data (like yours) and/or some simple notation beat a lengthy word picture every time, so please don't tell us what you have, show us.
I mean both final answers and mid-calculations are typically rounded even in textbooks.
If the relationship holds only for subset of $\\{1,...,N\\}$, then you need to indicate that. In this case you do not have the usual model. When you write $i=1,...,N$ you mean that relationship holds for each $i$. This is the same as writing $\\forall i \\in\\{1,...,N\\}$. Since $i=1,...,N$ looks nicer, it is used more frequently.
I added Bayesian because I'm currently working on a Bayesian inference model, and that's the context of the question.  (Plus I don't know stats well enough to pick any other keywords.)
I removed the new tags you added; they seemed to me either too general and not discriminative, or in the case of as.factor too narrow (since it's a function in R base). Please retag if you think anything was lost.
I may be confused here, but how does "the true value" fit into a Bayesian framework? Maybe you are referring to the posterior mode (or mean, or.. etc)?
Thanks a lot. I think that makes sense. It's interesting that I haven't seen any textbook mention that case.
For reference, I've seen quantile regression suggested when you want asymmetric loss functions, see [Berk, 2011](http://dx.doi.org/10.1007/s10940-010-9098-2), PDF [here](http://www-stat.wharton.upenn.edu/~berkr/loss%20copy.pdf).
@user39496: Once you have an answer you like, you can accept it by clicking on the greyed-out tick next to it.
Aptamer active motif selection, forest ground humidity forecasting, digit OCR, multispectral satellite image analysis, musical information retrieval, chemometry...
I admit I only very quickly skimmed your post, but isn't the fact that the differences in the right-tail converge to zero tautological? Would it be more appropriate to look at, e.g., the absolute relative error?
Your result is most likely a massive overfit. Test your model on an *independent* test set, not on the training set.
Your post doesn't contain a clearly defined question.
1) What is a `replicate`? Is it like a "case"/"observation", such as an individual in a survey - a unit of random sample? 2) Where is the place of the treatment factor? Do you have 30 "replicates" after the treatment and another 30 "replicates" as control group. Or what?
I'm curious what you mean by "fails."  Obviously, to get an adequate approximation to your spectral density you may have to increase $N$ (and then subsample the resulting vectors), but I don't see any fundamental reason why this shouldn't work.
Thanks dear to share it.
The design of the object oriented program would be nicer.
@IanStuart: "reading lmer output is currently problematic to me..." - Doug Bates has made a draft of an upcoming book on the lme4 package [available online](https://r-forge.r-project.org/scm/viewvc.php/*checkout*/www/lMMwR/lrgprt.pdf?root=lme4)
+1 It's also a good idea to consider whether similar explorations have been made with respect to other factors.  Other things to think about are (a) cross-validation or using hold-out data to confirm hypotheses developed by data snooping and (b) [Tukey's Honest Significant Difference](http://en.wikipedia.org/wiki/Tukey%27s_range_test).
Exactly what I was looking for. I do, thankfully, have equal sample sizes in all 4 measurements, so I'm investigating artificial pairing and may go that route. One follow-up (for now): You say that the t-test involves assuming independence among the 4 samples. Control and Intervention are definitely independent, but Control-EL and Control-BL may not be. How would you adjust this in that case?
As far I know principal Hessian directions are used for post-hoc data reduction & not tied to any particular experimental design. Don't think anyone can tell you what kind of design to use for your experiment without your giving more specific information about the problem. And there are whole books written on regression - exactly what trouble are you having? ("Conical analysis" I've never heard of.)
@Riyaz, could we use the following analogy to Factor Analysis to understand your question? Often someone will factor analyze a set of variables to cluster them into groups of variables that seem to 'hang together', but then the analyst needs to think about the nature of the variables that make up each cluster to come up w/ a name for / way of thinking about what each cluster (factor) *is*. Is that essentially what you're getting at here?
How are you calculating the shrinkage parameter of the ridge regression?
@Wolfgang Calculating correlations when you KNOW that there is likely a third variable that is causing both IS a bad idea.
I think that's bound to happen as you'll always have high volume vs high rate as factors in the overall rate, but if each sites level of contribution to the overall rate remains fairly static over time, this shouldn't be a problem. It's only where that rate changes across months that it will be scrutinised. One issue is that one site could have a huge numerical change upwards (but no rate change) which will obviously cause other site's contribution to the overall rate to reduce.
This is probably a bad combination of identification issues (too many 0s) and numerics. Why don't you stick with Stata if it works?
Older literature often delights in formulas of rococo complexity for this kind of thing. In practice I would lean over backwards to thinking of this as a regression problem so that the interpolated values are just the predicted values from the regression. See e.g. http://stats.stackexchange.com/questions/60500/how-to-find-a-good-fit-for-semi-sinusoidal-model-in-r/ The key principle is that the main cycle repeats once per year.
Thanks for clarifying. It's impossible to know in advance that a term like "epoch time" (a) has multiple meanings (depending on the operating system) and (b) may have completely different meanings for different people (such as an epoch time for a geographer). So to avoid confusion, multiple conflicting answers, or lack of answers, it's best to explain yourself whenever you can.
@MichaelChernick: That attempt to ping whuber won't work. He has to already be actively participating in this particular comment thread. :)
Normality of the outcome (dependent) variable is never an assumption in regression (except when regressing against a constant).  But, again: *why* are you performing this procedure? For estimation, prediction, forecasting, or what?
Ah alright, I get what you meant! Thank you!
Excellent suggestion and I like the simplicity of collapsing the data.  Assigning each value to the nearest known peak is what I was trying to do in my answer (which I have now deleted) but I complicated it greatly and unnecessarily by keeping to a row by row method.
If you are comparing groups then you should compute the confidence intervals of the differences.  The formulas are in most intro stats text books and are implemented in any decent statistics package (`t.test` function in R is one example).
I'd recommend transferring this question to GIS SE. The question is about how to apply [kriging technique](http://support.esri.com/en/knowledgebase/GISDictionary/term/kriging) and I believe the GIS folks can provide more technical advices to the questioner. Most GIS software such as ArcGIS or R (with the right packages installed) can perform this interpolation.
There's nothing stupid about the question. But you ought to sit back and think hard about the non-technical issues before getting into the numbers: do you really think you can generalize about the Germans, British, Dutch, or whoever, based on what a few people say on a *Web survey*?  This seems rather unlikely, but if you can find a way to support this expectation, then it would make sense to look at the numbers (at which point you should heed @Ladislav's advice).
The p-value corresponding to condition is the one that relates to differences in condition. I notice the variable 'weight' is actually spelled 'weigth' in your code. From bitter experience, this sort of misspelling in code can lead to a lot of misery, even if you're consistent about it.
+1. This gets to the geometric essence of the situation: because $M$ is an orthogonal projection whose kernel is precisely the column space of the design matrix, when the errors are iid with variance $\\sigma^2$ the expectation of the SSR must equal $N-K$ times $\\sigma^2$. However, somewhat contrary to your characterization, *we must understand $K$ to be the dimension of the column space* (which may be smaller than the second dimension of $X$). Knowing some kind of *formula* for $M$ is immaterial; these are the only properties needed to obtain the result.
Your interpretation of that sentence is correct: the author is saying that boosted models are not prone to overfitting. What else specifically do you need clarified?
That might be a difference in what practitioners call their analysis, but nothing forces a system of structural equations to be parametric. 

@zkurtz:
There is a long and technically detailed discussion of what SEMs are in Pearl's Causality. If you don't have the book I could attempt to post a brief summary and track down the example he refers to in the link you posted.
I mean you haven't given us any information about what call you actually made and what the data that you fed it looked like! It's very hard to debug code that you haven't seen.
@Aaron thank you for pointing that out. y= [0,10]. Fixed.
Please write the regression specification using mathematical notation and not computer language -a lot is missing, for example it is not clear the lags of what variable(s) are included in the regression. So: mathematical notation. Also you must specify the units in which each variable (dependent and regressors) is measured.
thanks, that's handy :)  I know trying to second guess the future has limited use, but hopefully it will have some motivational benefit, as it does prove things are on the up and not sliding down :)
Why do you want the summary statistic?  What's its purpose?  How will you use it?  To me it seems that *that* will determine what is an appropriate summary statistic.  It could be that given your purposes, there is no particular *mathematically* justifiable function, but you can discover empirically (in some sense) what weighting to use by comparison with the things you want to learn about through the summary statistic.  For example, if you chose to weight linearly, maybe it would make sense to perform a regression to determine weights.  Or maybe that would be nonsensical. Depends.
Looking at this document, http://support.sas.com/publishing/pubcat/chaps/59498.pdf, their also appears to be differences in definitions between SAS and SPSS that could cause confusion. Formats in SPSS appear to be referred to as informats in SAS. Value labels in SPSS appear to be referred to as formats in SAS.
@stefanos: Can you please tell me the name of the application as I am always interested in pursuing software solutions dealing with this issue. You can email me at my contact info.
What have you tried? Have you searched elsewhere on this site to try to find an answer? Also, if this is for homework or self-study, you should add the `self-study` tag. See http://stats.stackexchange.com/tags/self-study/info
+1 Very nice!  This problem *had* to require numerical integration (except for special cases such as all means being equal), so the availability of this package and the reduction of one in dimensionality are both useful contributions.  By the way, as a check you can compute P(Y>X and Y>Z) = 0.3200386 and P(Z>X and Z>Y) = 0.5353126; the three results sum to 0.9999999, which is reasonably close to 1.
@andrea, you are probably correct, I was focusing on the "Overall" part rather than thinking logrank vs liklihood ratio.  Either way I would not interpret it the way the original poster wants to without a lot more information.
@heropup $Y_1 = X_1$.
I suspect that kitty implies $S$ observations drawn from the population of size $N$, rather than $S$ samples of whatever size. I.e., for her, "sample" means one case, one observation, one line of data.
I would maybe start with a multinomial regression in glmnet, since that will be pretty fast.  If you don't get good results, look at a RF/SVM/some other method of discrimination.  IMHO, you're going to have some trouble distinguishing 100 classes.
I suppose this is the full citation to the article @Will is referring to: Berk, R. A. (1983). An Introduction to Sample Selection Bias in Sociological Data. American Sociological Review, 48, 386-398. doi:10.2307/2095230

There are several freely available versions of this paper, which you will find on Google Scholar, e.g..
I've edited your post to remove the anova tag (which may not be the method of choice) and added an econometrics tag.
Continue:: I think this is making a bias, and by only CENTERING Y, and not standardizing it, we will avoid the bias. What do you think?
Hi whuber! I need a suitable implementation e.g. in Pascal. For moderate N (like 20) the well-known Irwin-Hall CDF formula is no problem at all, but I do not want to spend too much computing time, e.g. for an integration, (inverse) Fourier transform or whatever. Of course, the Fourier transform approach is elegant, but not so accurate, because I am highly interested in CDF(x) for "large" x, so the tail area matters for me!
Kind of "nearest neighbor" classifier. Thanks for the answer Marc :)
From what I can see, there's a fundamental conflict (or inconsistency) underlying this question. People use the EWM when they do not care to analyze the data to characterize and quantify the serial correlation, but in order to answer this question the serial correlation *must* be estimated; but then why would you use the EWM in the first place?
could you first describe your data and models?
Hey Randel are you still there? Do you know why SPSS GEE doesn't run when duplicate cases exist? If it is going to be repeated-measures, it has to expect duplicated cases. But it keeps giving me this error: "There are at least two records with the same values for the subject and within-subject variables. No output will be displayed.	
Execution of this command stops." From studying other forums and Googling, I know this is caused by duplicated cases in the sample, but wasn't it supposed to be "repeated-measures"?! So why it cannot tolerate duplicate cases? Do you have any idea?
@Glen_b My second reading was that Harry missed a class **because of** political unrest in his country.
There is an R package for the Bradley-Terry model. Look on Rseek.
About visualization techniques, there are some discussions lurking around there: [Visualizing Likert Item Response Data](http://stats.stackexchange.com/q/3921/930). For raw data, I believe having a short overview should be enough, although it is possible to [rely on external web hosting systems](http://meta.stats.stackexchange.com/q/1035/930). Now, the question is what the graphical displays and numerical quantities aim to demonstrate/illustrate.
Estimating a Markovian transition matrix?
I suspect you need to model the visitor, then use queuing theory to do your analysis.  How detailed your visitor model is is up to you.
By "the Jose and Douglas book", do you mean Pinheiro and Bates' *Mixed-Effects Models in S and S-PLUS*?
@Isaac If I have understood the context correctly it seems we "know" nothing: all parameters are estimates from data.  Thus, at a minimum, we would want some information about the standard errors of these estimates.  If I'm wrong and you know the means and covariance matrix for certain (or have highly accurate estimates), then the problem is one-dimensional: it comes down to separating two normal distributions on the line, which is straightforward to do (this is the basis for power calculations when testing equivalence of means, for instance).  Would that be what you're looking for?
It may work; normalization can be thought of removing the units from the equations. However, the result will heavily depend on the preprocessing, and you will get fundamentally different results depending on how you preprocess your data.
@Tusell. Thx for the reply. I'm going to have to check out the book you pointed out. Not easy to find books that put it all together like that.
(1)have you looked at the residuals, done a lag1 test (Durbin-Watson), white-noise test (portmanteu) or looked at auto-correlation plots? (2) if this is OLS then an assumption that the residuals are not autocorrelated, in this case where there is high suspicious one should either prove they aren't or correct (3) there are a very large number of methods to correct/account for autocorrelation including praise-winston or cochrane-orcutt (if this is OLS)
I understand what you're saying, but when you say "basically the same", that's where you're going astray. Density doesn't work that way, since the transformation "stretches the density", and you have to account for how $dw$ transforms to $dy$ (literally, $dw=\\frac{1}{y} dy$ in your problem). You *can't* simply replace variables in the density like that, you need also to take the element $dw=\\frac{1}{y} dy$ across. It (that replacement of $y$ by $\\exp(w)$ when $Y=e^W$) can, however be done with probabilities of *events*, which is why you can do it with the CDF.
bean machine by the author of the package animation... http://yihui.name/en/wp-content/uploads/2010/07/animation-useR2010-Xie.pdf
It might help to post the `head()` of your data.
Prior parameter $\\Theta$ has density $p_\\Theta(\\theta)$. if the realization of $\\Theta$ has value $\\theta$ while $x$ is the observed value of a random variable $X$, then the value of the likelihood function $L(\\theta\\mid x)$ is _precisely_ $f(x\\mid \\theta)$, the value of the conditional density $f_{X\\mid\\Theta}(x\\mid\\Theta=\\theta)$ of $X$. The difference is that $$\\int_{-\\infty}^{\\infty}f_{X\\mid\\Theta}(x\\mid\\Theta=\\theta)dx=1$$ for _all_ realizations of $\\Theta$. However, as a function of $\\theta$ (and fixed $x$), $L(\\theta\\mid x)$ is _not_ a density: $$\\int L(\\theta\\mid x)d\\theta\\neq 1$$
Not sure, but maybe sparsity of an estimator (vector valued) $\\hat{\\theta}$ means that most off-diagonal entries of the covariance matrix are zeros, so components of $\\hat{\\theta}$ are mostly orthogonal. Maybe you should look for "orthogonal parametrization" in the literature.
@nikosdi since both the question and answer are different from what you are asking for, I think a new question would be better. Both forms of the question could be useful for people.
A very good approximation for $\\ln(1+\\exp(x))$ for $x>7$ or so is $x+\\exp(-x)$. This means you can readily take logs of numerator and denominator and work on log-scales when dealing with large arguments.
@tom: By the way, I *was* mistaken: in 2D $\\mathrm{Var}(z_i)$ is 1, it is $\\mathrm{Var}(z)$ that is equal 1/2. I have updated my question with some simulation results. Seems like the correct formula is $1/D$.
If you have lots of data I think joint densities are more powerful.
Could choose the smoothing bandwidth using cross-validation. What's interesting is determining whether the dip's real - an artefact of the experimental protocol or an interesting fact about bird behaviour - or just overfitting, as @Underminer suggests it might be.
Please clarify the exact model you have, as it's not clear given your description why you need a prior on the Dirichlet.
imagine that you had a set of points (x,y) and you were considering whether you might fit a straight line or a quadratic. If you compared the RMSE, the quadratic *would always beat the straight line*, because the line is a quadratic with one parameter set to zero: if the least squares estimate of the parameter is exactly zero (which has zero probability for continuous response), it's a tie, and in every other case the line loses. It's the same with the Poisson vs the negative binomial - a free Negative Binomial can always fit at least as well as a free Poisson.
This seems like reasonable advice, although I would be concerned about two particular aspects. 1) Rules like X standard deviations shouldn't be relied upon, the metric should be evaluated upon out of control sequence length, false positives and more general resource constraints. 2) I'm not sure if this is sufficient to account for the upward linear trend in the data, although since the objective is to see "too low counts" perhaps that is not a big issue.
Angels? Surely not. Fischer whispers to you in your tea as you're making it. And if it's really late at night, Kolmogorov and Feller shout at you from your vodka glass.
I had a second thought. I’ll post it...
Hi, thanks for answering. I'm new to time series so I'd appreciate it if you could expand on why I might be more interested in the variance of the residuals. I have started with (what appears to be a non-stationary series) and after differencing now appears stationary. So in this case are the residuals and why would the variance of those be of more use/interest than the stationary series as a whole?
As I see it the issue with Mallow's CP, AIC, and BIC is primarily that they are unfamiliar to most end users (again, this is not targeted at the statistical community). You are asking people to use something new and strange, and for the most part they simply won't understand. There is also the issue of comparing models between different application in which cases something like AIC is useless.

Adjusted $R^2$ will under-penalize additional parameters so has over-fitting issues (so does Honest $R^2$ but to a much lesser extent).
Try answering this: My neighbor has two kids--you know that one of them is a boy. What's the probability that she has two boys.
Or, you can do both of the above.  The Q-Q plot tells you where and how substantial the deviations from the model are.  The K-S test (which is based on the supremum of $\\hat{F} - F$) tells you whether that difference can be reasonably attributed to sampling error.
The optimization problem is no longer a QP because your constraints are no longer linear in $\\alpha$. Like I said previously, that result has no interpretation like standard SVM so it probably won't behave like you expect it to.
I'm getting the same error you are, and it's driving me crazy. I think it has something to do with the number of vars with respecto to the number of observations but I'm not sure. Any suggestions would be greatly appreciated.
Thanks.Then I should look for P[X>=40] right? As there is a total of 40 days.One question.Should my null hypothesis change into **H0:lambda=30** or is it lamda=3
http://www.inside-r.org/packages/cran/lme4/docs/pvalues