tttjjj/elt_bench huggingface dataset:
That changes the question substantially, John, because in its current form it explicitly states otherwise: it asserts that only 5% of the values are missing. Even if we understand the 5% to apply to all *entries* in the data matrix, rather than 5% of the *cases,* anyone taking the example as indicative of the nature of your data would validly conclude that no more than 10 * 5% = 50% of the cases have missing values. The three things that are most important to describe in such questions are (1) the purpose of the analysis, (2) the nature of the missingness, and (3) the amount of missingness.
Look up "finite population adjustment"
+1 | I think the first sentence implies that the application of outlier detection is entirely subjective and therefore I vote for keep as is.
Thanks for the update about the homework status.  If you need more specific help, let us know.
Can we extract trend from a non seasonal time series data using STL, Census (X-13-ARIMA) and classical decomposition ?. I thought all these require seasonal component i.e., frequency > 1.
@whuber, It is definitely not a conventional interpretation, should I call it something different? I'm just assuming I wont have to worry about improbable results happening. To your other question, I am asking why regression would be better.
I used ('ElemStatLearn')("klaR")("caret")package. #rapaio, what I really want to do is to see the overall model performance. Something like I can plot .This is because I want to put it in the report. How to do it?Thanks.
A terrible book, written by someone who does not understand statistics...
(Especially, after reading the pdf you recommended me)
Can you provide a reproducible example?  I think you could probably use singular value decomposition to decompose the Hessian into singular and non-singular terms ... (or equivalently [???], compute a generalized inverse, as in the `ginv` function of the `MASS` package in R ...)
Thanks for clarify my question. The problem I'm having is how to do the integration. I tried to expand and group $u_i$ together but no luck. I can't get familiar form. Anyone has idea?
@Michael This is not at all hard to reproduce, knowing that including lots of irrelevant variables will increase the overall p-value. Here's a case with up to five variables where you can watch the overall p-value balloon as each variable is added, while the individual p-value changes little. Try this in `R`: `set.seed(17);
p <- 5;
x <- as.matrix(do.call(expand.grid, lapply(as.list(1:p), function(i) c(-1,1))));
y <- x[,1] + rnorm(2^p, sd=2);
temp <- sapply(1:p, function(i) print(summary(lm(y ~ x[, 1:i]))))`.
QQplots are not recommended for discrete distributions. There is a HUGE literature about dealing with rounded observations. Use also "grouped data" in your keywords. [link](http://lmgtfy.com/?q=goodness+of+fit%2C+grouped)
Here's two interesting articles on scale-setting of latent variables: Gonzalez & Griffin (2001): *Testing parameters in SEM: Every "one" matters* (http://www-personal.umich.edu/~gonzo/papers/sem.pdf), Little, Slegers, & Card (2006): *A non-arbitrary way of identifying and scaling latent variables in SEM and MACS models* (http://www.agencylab.ku.edu/~agencylab/manuscripts/(Little,%20Slegers,%20Card,%202006).pdf)
**Better for whom or what?** Measuring the coefficient of the law of gravity in a lab? Reporting the profit of your company to the tax office? Correcting for instrument error?
I found that out too.  This is exactly what I implemented and it worked excellently.  I came here to answer my own question but you beat me to it!  Clustering was what I was trying to do.
Do you perhaps mean "double exponential" instead of exponential?  http://en.wikipedia.org/wiki/Laplace_distribution .  When errors have a double exponential distribution it is optimal to minimize the sum of absolute values of residuals, leading to median regression.  (Normally one would not model "errors" with an exponential distribution because an "error" should have zero expectation.)
@Pardis- Thank you very much for such quick response. Yes, I have looked at the sample guide of LIBSVM. Now I will look for the book you have mentioned.
What if one single class in one of the two schools has an extremely good teacher so that kids in that class will all be over the average of the other classes? The overall mean score of that school may be higher but just because of that class, not because the general population of that school is better.
I just hoped I was doing it right because the results were very good (too good to be true)!! Thanks for you help!!
Excellent explanation, @whuber. Thank you.
A first start would be to normalize each series by subtracting the mean from each data point and then dividing each point of the resulting series by the standard deviation. Do this separately for each series and then plot them together.
The easiest way is to compute the expression $Pr(q>p|xynmI)$ for various values of $x,y,n,m$.  I'll add a table if you like.
Nice answer, cardinal! I was wondering [what kind of optimization problem LAR is solving](http://stats.stackexchange.com/q/55404/1005)?
Why do you eliminate the data points after you do LOOCV?
Can you be more specific about what you want to know? Are you wondering how ordinal logistic regression works, or how to visualize ordinal data, or something else? I also suspect that the answer you are after may well already exist on the site, if we knew where to look.
"*some of the dependent variables are not normally distributed*" -- good thing that's not an assumption of ANOVA, then. There is a normality assumption, but it's not an assumption about the unconditional distribution of the DV. Specifically, the errors are assumed to be normal. What do residual plots look like?
Consider each eigenvalue/vector pair as approximate covariance matrices $ \\lambda_i e_i e_i^T $ Then the correlation in this matrix is actually the correlation between the eigenvector and the other original values, these may be negatively or positively correlated, so if you remove an eigenvalue that is  negatively correlated with an original variable, you can cause an 'increase' in the correlation between different original variables.  Although the maximum change in any direction for a covariance matrix element is $ \\lambda_i $ per eigenvalue/pair removal.
The beta distribution illustrated above has an **antimode** at $x =  0.5$. This kind of minor terminology polarises people all the way from "yet another term we don't really need" to "just the term I was looking for".
I think that is what Nick Cox was trying to emphasize. The estimates for poisson model by be accurate, but the p-values and SE from such a model are bound to be to be incorrect since the data doesn't fit the assumptions of the poisson model. So you need some way to adjust those p-values/SE to provide a more appropriate estimate: quasipoission or sandwich estimator are two different approaches to achieving this end. Quassipoisson is just easier in R and sandwich estimator easier in Stata. (don't have much useful to say about the iterate numbers warning, sorry)
@chl I'm honestly a bit lost with this, but I've updated the question with an example of what I'm trying to achieve. If that's not enough I'll try to expand it a bit more.
So you suggest equivalence testing to infer which pairs did not contribute to the global difference revealed by an ANOVA test? Interesting thought! In the end it's a completely different approach to the same interpretation "pairs A and B contribute to the revealed global difference, but pairs C and D are to similar". But I'm not sure if it is correct.
Taylor: $e^y=\\sum_{n=0}^\\infty y^n/n!$. Take $y=e^t\\lambda$.
I had tried it but this does not solve my problem. I added some examples to show my problem.
"Moderator" generally means an independent variable that interacts with another to affect the dependent variable.  Therefore, if model 3 has a moderator, it needs an interaction term.  I don't see why you would have a model 3 separate from your model 4.
@Adhesh Josh: To clarify a small but important point: study hours is one variable; exam marks is a second; and location (home vs. school) is a third. You have three, not four variables in this sentence.
@BrashEquilibrium: "this is freaking geology" -- I just thought about that line again and it made me laugh.
@Tim: you wrote $x_i \\in \\{0,1\\}$, but it should be $\\{0,1,…,n\\}$.
For each of the measurements, do you know the ordering? Ie, do you know which of the 16 pre-treatment blood samples was drawn first & when, which 2nd & when, etc? Then all measurements could be placed in time.
@Guy By analogy, we could characterize chemistry (another "mathematical discipline") as asymptotic distribution theory and C* algebras. Doing so is nominally accurate but so completely misses the essence of what chemistry is and its aims that no chemist would recognize it. Similarly, contrast your characterization with what [leading professional societies say statistics is](http://www.amstat.org/careers/whatisstatistics.cfm): they are worlds apart. "The science of learning from data, and of measuring, controlling, and communicating uncertainty." Not one mention of probability there.
Have you considered [R](http://www.r-project.org/)?
Terminating the limit in the integral at the range of the data (for the Normal approximation to the Binomial distribution) is unusual, but it makes no practical difference in any application where this approximation is reasonable: the difference between the two calculations is (much) smaller than the error of approximation. Usually, then, one *does* integrate to infinity, because such integrals are either already tabulated or are directly calculated; integrals between two finite limits are calculated as differences of two such tail areas.
How can i get the appropriate picture of y-intercept parameter ,$\\beta_0$ , if the data are `ht <- c(169.6,166.8,157.1,181.1,158.4,165.6,166.7,156.5,168.1,165.3)
wt <- c(71.2,58.2,56.0,64.5,53.0,52.4,56.8,49.2,55.6,77.8)`
and `lm(wt~ht)` ? I have edited the graph of this data.
Then why don't you ask him to send you the reference for the putative "correction"?
Why not compute the ratio, or the difference, and model that time series, forming whatever kind of interval you need, as you would for a univariate series.
@AzulaR., the question is whether intervention has an effect on the prevalence. I guess from the two ways you mentioned the first one seems more relevant.
There appears to be a freely available version of the paper here: http://webdocs.cs.ualberta.ca/~nray1/CMPUT615/Zip/Deep_NN.pdf
@Glen_b Yes it helps. John's comment was helpful as now I  know that the values are derived from the standard normal distribution.
a week later, I have figured out more precisely what the problem was.  I was using two factors, admitting school and admit-year.  For some years we only had data from a single admitting school, so the levels for that year and that school were redundant.  Understanding that makes me comfortable having them drop out of the model.
I looked into parallel plots, and it seems to be what I need to do. Thank you!
Closely related: http://stats.stackexchange.com/questions/13314.
Yes the data represents intended touch points which a user was aiming to touch on a smart phone. The response was the actual touch location registered by the phone.
I'm trying to build a regression model which can predict the intended touch location from the recorded touch location.
@AlecosPapadopoulos Thanks ... so applying this to WLS, it is important to test for any remaining heteroscedasticity, eg by examining the WLS residuals, rather than automatically opting for robust standard errors.
@Aksakal On what basis? (No pun intended). Splines are a useful and widespread tool for nonparametric regression and the example has been taken from a classic statistical text book.
"_Within each year, there is data from over 100k locations._" Do you actually observe all locations or just the average value based on them? If yes then yo can go for panel data models as @crayola suggested in linear context. Though some special ecological models as @GaBorgulya mentioned may require much less information for the parameters to calibrate rather than estimate.
Is plotting it a requirement? There are pretty good ways of estimating the factors without plotting.
I would but I think in all cases this is really all a matter of choice as to what you think is the best display of what you want to show.
are your inputs and outputs discrete valued or continuous? How many different values can your inputs/outputs take?
One might be able to derive nearly or barely significant from a Fisherian perspective on p-values where they're taken as evidence for an effect.  However, even that is weak because significance itself wouldn't be defined anymore.  Either something is significant or not in a t-test.  It sounds like you're hinting at the idea that the difference between significant and not-significant is not itself significant.
Thank you for your answer Behacad! Indeed, the reason I'm asking this is that the statistical significance of the analyses is not that clear. There was a statistically significant difference only in one group, where t(29) = 2.16, p (2-tailed) = .039. Bonferroni correction would make this result non-significant, even if I used one-tailed p (.020).
I wouldn't agree. Asking for a data analytic algorithm and/or theory is in tune with our site. Particular programming questions should go to SO.
This doesn't really answer the question. Backpropagation is a method of training the network, it isn't how the network recognises images.
So does that mean you want to change your question to 'how do I do multiple comparisons after an ANOVA?' or do you actually want to know something like 'how do I do something like an ANOVA and multiple comparisons but without assuming normality?' ...? (Incidentally, what are your sample sizes?)
If you have any advice for the last question (courses over image format, metrics).
In this particular case, I do know the exact age distribution of the underlying distribution. Or at least something close to it. But this is an interesting point in general.
Because this answer focuses on a limited and specialized version of the question and uses the key terms "estimator" and "statistic" in unconventional ways, without alerting the reader to that fact, I worry that it may mislead or confuse people.
Nice reference: thanks.  It comprehensively covers everything mentioned here.
Just for the record, the population sampled from here is finite; so if you sampled a large fraction of it you'd need a confidence interval on a hypergeometric proportion rather than a binomial. For 4k sampled out of 400k of course it's not an issue.
A very closely related thread at http://stats.stackexchange.com/questions/5347 addresses this question for the case of more than two binomial variables (with a focus on a much larger number of them). Of course all the exact solutions offered there apply to this simpler case!
Your question isn't all that clear, but it does seem like you have data collected over time. If this is the case, regression is not appropriate. You need to account for the nonindependence of the data. Two general sets of methods are time series analysis and mixed models (aka multilevel models, hierarchical linear models and other terms).
See [this question and its answers](http://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs). Does that answer your question? If not, please say what else you are asking about
Henrik - when I try this I get the message: Error in terms.formula(formula, "Error", data = data) : 
  invalid term in model formula
For rep I have 1,1,1,1,1,1...10,10,10,10,10,10.
For season 1,2,3,4,5,6...
For treat 1s and 2s.
dv is a continuous variable.
Im not sure where im going wrong now - is there a way to post a datafile on here?
Hmm right that makes sense. So if I wanted to come up with a single logistic regression model that is equivalent to the average of the output from these N bootstraps (i.e. a single set of regression coefficients) how would I do that?
Support vectors are those data points that participate to the solution, i.e., define the curve that fits the data. I don't think there is the notion of hyperplane in the same sense as classification, you could say that it is the curve itself.
Check out this video lecture by Teh himself: http://videolectures.net/mlss09uk_teh_nbm/

At about minute 43 there are some equations that I think are really helpful!
It is important to note that "dimensionality reduction" methods typically do *not* maintain "relative distance information." Whether they do or not depends partly on the method and partly on the intended "distance."
Two ideas to investigate: Partial proportional odds model and (in the multinomial analysis especially) combining the bottom two categories.
Just to elaborate: each measurement of parameter ($X_1$,$X_2$,...,$X_{100}$) has an assigned chance of being a false positive ($P_{f1}$,$P_{f2}$,...,$P_{f100}$). 

So how to measure $\\sigma_X$ is complicated by the existence of these false positives.
All you need to understand is that multiplying *anything* by a constant value is tantamount to changing its units of measurement. That immediately shows that the rates of change (the derivatives) get multiplied by the same constant value. This principle applies term-by-term in the sum.
More details would do no harm, but on this information either non-linear least squares or fitting a quadratic to log y as a function of x.
+1 This is a good summary of the paper's main point.  I especially appreciate your attention to the distinction between an estimated p-value and the true permutation p-value.
Spearman's rank correlation seems appropriate (as probably any other rank correlation).
@John, I'd say that the bootstrap is an overkill in this simple and classical situation. See my answer.
"No, I do not think so." But you wrote exactly that. "2) (X,Y,X) -> you state and I do agree X|(Y,X) Dirac and (X|Y)|(X|Y) you state normal, I say Dirac." **Whaaat?** Could you please indicate where I would have used the notion or the notation (X|Y)|(X|Y)? Since I never have, where did I "state normal" (your words)? Since I nowhere did so, what are you talking about? For some accurate and precise statements destroying any hope that the notation (X|Y)|(Z|Y) could be anything but a meaningless string of characters, please see my post. Oh and just in case: *read slowly*.
Andy's answer is excellent here.  I *think* this discussion helps formalize the idea of 'regressing to the mean' which crops up in a bunch of places: http://en.wikipedia.org/wiki/Regression_toward_the_mean
Also, it's necessary to keep in mind that the comparison of LandUseLow to baseline, in the presence of continuous variables, compares LandUseLow to baseline when these continuous predictors are 0. Often, this is not reasonable, but I can't tell from the variable names.
It appears one of the questions on the right is very similar, http://stats.stackexchange.com/q/173/1036 , care to elaborate on how that does not answer your question?
I don't see how the sediments are important here. Aren't you counting the marbles? Are you tossing the marbles back?
@Stephen In the example he changed 2 to 5: only one value was altered.
`dnorm(x)` gives the density function $\\varphi(x) = \\frac{1}{\\sqrt{2\\pi}} e^{-x^2/2}$. If you want the density for mean $\\mu$ and *standard deviation* $\\sigma$, then you can pass these as the second and third parameters, respectively. Use `pnorm` in the same way to get the cumulative distribution function. `qnorm` gives the quantile function (inverse cdf). Typing `?pnorm` will give some more informatio on all three.
Cross-posted at http://math.stackexchange.com/questions/258761/expectation-of-an-exponential-function.  Although questions like this are (marginally) on topic here, this is a purely math question that has been asked and answered on that site, so I'm closing this copy of it.
@ John: Are you sure about that value of n? This example seems to say otherwise. http://stats.stackexchange.com/questions/71525/critical-effect-sizes-and-power-for-paired-t-test. Although I guess I didn't really say that my value of alpha is 0.1. Were you thinking 0.05?
(ctd)... So if $X$ is the log-scale variable, but they want to talk about the mean of the original variable, they're interested in estimating $\\text{E}[e^X]$.
I wonder if the SVM aspect isn't irrelevant here. It sounds like you just want to know which subset of variables out of a total of about 100 are relevant for determining whether a protein is a "Go" or a "No-Go". Is that about the whole of it? If so, you are interested in [tag:feature-selection].
I was suggesting forgetting EM & using any old method to maximize the likelihood function of all three parameters directly, checking graphically that the likelihood function is well-behaved. You can get at starting values by comparing kernel density estimates of the real data with simulated ones. (There are probably better ways, which I why I didn't put this as an answer, but brute force should work for this simple case.)
Oops! Thanks for offering me the opportunity to correct! Your first understanding was correct; I meant for the total hours in Time 1 to be 10000 and Time 2 to be 12000; I have now edited.
Thanks for the detailed and reassuring answer!
@PeterEllis: Post-stratification weighting seems similar, but I'm interested specifically in the multilevel case here. An important part of my question was missing, I have updated it.
A few questions: How many components do you have? Besides the sample size, is there anything that makes the display of this PCA output need to be different from the display of other continuous variables that one might be dealing with? Are you trying to contrast scores of different groups, and if so how many?  Generally, what are you hoping to achieve with your displays?
(1) logistic regression with k-1 dummy variables for your k bins. (2) there are number of alternative ways to include non-linear relationships in logistic regression. Restricted cubic splines being one. But dummy variables should be a reasonable start
Your answer is not answering the actual question.
@whuber, I also though about this possibility of having just one normal r.v. $\\xi$, the rest are just linear transformations of it like $\\mathbf{X}_i = a_i \\xi + b_i$. May be I should start from this simple scenario.
How can i post the image of same ?
Examples of better plots?
@rpierce I'm sorry if I made my question a little difficult to follow. $\\overline{err} = \\dfrac{1}{N}\\sum_{i=1}^N L(y_i,f(x_i))$ is comparing the fit of your model to the data used to fit it. So for squared error loss that would just be $\\dfrac{1}{n}\\sum_{i=1}^n (y_i-\\hat{y}_i)^2$
I don't think there's a need for bounds, it should be possible to evaluate this probability exactly. What's the distribution of $kX-Y$ for some constant, $k$?
If you estimate the parameters and don't want to use a prior (different then using an "uninformative prior") then you essentially just want to do [Maximum likelihood estimation](http://en.wikipedia.org/wiki/Maximum_likelihood), i.e., $\\theta^* = \\arg\\max_\\theta f(X|\\theta)$.
@naught101 Perhaps my understanding of false positive is false (har har). What the model does NOT produce is an incorrect solution to a question that it labels as correct (e.g., erroneously producing an output of 5 to the question what is 2+2?). The model only produces an output if the output can be formally verified. This is similar to a mathematician generating a solution to a problem and then checking their work to see whether their solution is mathematically sound. If the model cannot generate a solution in this way within the time limit, it simply doesn't generate any output.
I might not have made my question clear. You're right that there are many transformations that could change one picture to another and for that reason my regression would be very noisy. Nonetheless, I would like to treat each pixel as a data point and compare it to the corresponding data point via a regression. The images were created by trying to copy a model image. Any idea how to do this regression?
thanks helpful!