tttjjj/elt_bench huggingface dataset:
+1. But you might want to consider something a little more general, such as some form of line plot. After all, as soon as you can draw a line segment, you can draw *anything*. This echoes the idea behind "Hello, world!": once you can get readable output from a program, you know you can communicate with the computer and the rest is just details :-).  (The details can be *hard*. When learning to write Assembly code for the IBM 360 mainframe, the simplest way to get output was to store it in RAM and create an error at the end of execution, causing the system to print a hexadecimal core dump!)
1) SPSS copies the entire dataset to the new dataset, no other way round. Does that really bother you? 2) Dont't quite understand what you seek. ID variable is necessary both in the original dataset and in its copy, set2, to be able to insert back missing cases from set2 into the original set. As for creating custom function out of that syntax for future, you could create a macro (read DEFINE chapter in SPSS Command Reference Guide) or write command extensibility function in Python language which is an add-on to SPSS.
Also look here: http://stats.stackexchange.com/questions/29932/how-to-generate-predictions-with-rjags/29946#29946
(ctd)... Perfect independence is in many cases an illusion, a convenient approximation to the very slight dependence that is the rule, not the exception.
One thing that might help is to center $x$ *before* creating your squared term (see [here](http://stats.stackexchange.com/questions/29781/when-should-you-center-your-data-when-should-you-standardize/29782#29782)). As for the interpretation of your squared term, I argue that it's best to interpret $\\beta_1x_1+\\beta_2x_1^2$ *as a whole* (see [here](http://stats.stackexchange.com/questions/28730/does-it-make-sense-to-add-a-quadratic-term-but-not-the-linear-term-to-a-model/28750#28750)). Another thing is that you may need an interaction, that means adding $\\beta_4x_1D+\\beta_5x_1^2D$.
But let's say that some products are vastly more popular than others. So, a little known movie comes out that only tree people have seen, so it has 0s in all of the other columns, not because the other people didn't choose it, but because they haven't been exposed to it. By contrast, everyone has seen Lord of the Rings, and just because someone watched it doesn't mean that they're "the type of person" who likes fantasy adventures based on books. They just watched because it was popular. How could you account for this to recommend the new item, instead of the blockbuster?
I assume you want to use `dcat` as a prior. In your model specify e.g.: `mu ~ dcat(pi)`. Then specify `pi <- c(0.2,0.5,0.3)` and include it in the `jags.model` data list: `data=list(pi=pi, ...)`.
14 predictors for the regression is fine. After that I run relative importance analysis, and bootstrap CIs of the RelaImpo estimates. These CIs are pretty large (due to correlation structure of predictors and the way they are bootstrapped). N=2k is not large enough for this analysis.

The missings are weird. There is about 25% dropout from week 0 to week 12, and week 6 seems to be just a smaller sample (reanalysis don't know why that is). 

I don't know of any other way than listwise deletion in regression analysis with multiple timepoints. What would you suggest? Thank you
It's unclear what you are seeking here. If you have a specific question about facts on dialysis, this isn't the forum. If you have a conceptual question about "lifetime risk", please spell it out. Meanwhile consider that people's lifetime risk of contracting a fatal condition is 100%, but that doesn't mean that everyone is just about to die. Similarly, you or I may be in the 3% who need dialysis in their lifetimes, but that doesn't necessarily mean right now. In this particular case, there may also be limits on the availability of dialysis facilities.
Transforming "a linear combination of the input into a non-linear output" is a basic part of the *definition* of a [Linear Classifier](http://en.wikipedia.org/wiki/Linear_classifier#Definition). That reduces this question to the second part, which amounts to demonstrating that Neural Networks cannot generally be expressed as linear classifiers.
You can actually get `qqplot()` to do all the `sort`/`length`/`approx` calculations for you:  `d <- as.data.frame(qqplot(x, y, plot.it=FALSE)); ggplot(d) + geom_point(aes(x=x, y=y))`
There's nothing wrong to [answer his own question](http://meta.stackexchange.com/q/12513/150510), Michael.
Related references of interest, Residual-based Shadings for Visualizing (Conditional) Independence ([Zeileis et al. 2007](http://dx.doi.org/10.1198/106186007X237856)), [PDF here](http://statmath.wu-wien.ac.at/~zeileis/papers/Zeileis+Meyer+Hornik-2007.pdf), and another thread here on [visualizing contingency tables](http://stats.stackexchange.com/q/4211/1036) with a few references. I believe the Zeileis article has a nice discussion on color that may answer your last question (it may be good to peruse the references to see if they cite the chart you are talking about as well).
Thanks, my question was more theoretical on actuarial survival analysis without competing risk rather than focused on the KM estimator. BTW your book is one of my favorite (I read the first edition when I was a student). Pity to see GP6 now has....wait for it... **pie charts**. Just joking.
'str(ozone)' displays the structure of the ozone data set. In particular, you can see that the variables are 'Station', 'Av8top', 'Lat', and 'Lon'. The error message says that 'lm()' does not expect 'ozone', which is none of the variables, to be the response.
$\\varepsilon_{i1}$ is the error (noise) term on the $i$th subject after the intervention. Your final supposition is not what you do, no. I explained you would perform a t-test.
yes, it is homework. I will indicate homework next time. I am using R and using ltsreg compared to the least square model.
There's almost certainly an analytic solution here. You might have better luck over at the math stackexchange. Those folks probably have more collective experience with graph theory than us stats folk.
If you post an example of the actual functions or code that are resulting in differing results, we (or StackOverflow) might be able to provide more specific answers, although @ocram's answer is correct in general.
You don't transform residuals, but the variables (in this case, the response) - and the help on `boxcox` clearly says that what you need to pass to `boxcox` is a model, either as a fitted model object or a formula. But before you do that, tell us about what your variables are (especially what is your response measuring?), and what the problems are in your model diagnostics (what makes you say it's non-normal? what does it look like?).
@chl: Yes, why not.
@whuber I agree that the questions are related, but this one is different in that I am asking explicitly what the probability distribution equivalent of rotation would be. Essentially I would like to discover how far this analogy between probability distributions and distributions of mass can be taken, and what use can be made of this.
Huang and Du were by no means the first to come up with this idea.  Setting the penatly in proportion to the size of the class has some theoretical justification, however it breaks down with finite samples, so it is better to choose the weighting factor by cross-validation (or minimising a generalisation bound etc.)  However, in this case, I don't think this is what was meant by weight, but instead is to do with the confidence of the sign of the label.
I would close it as duplicate http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros/1445#1445
What makes the data points correlated?
@qftme let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/782/discussion-between-shabbychef-and-qftme)
@vqv Yes, sample covariance matrix. Original post is edited to reflect this.
Because the wording is important in this setting: seasonal patterns and noise are distinct "building" blocks in time series. Often one models trend, season and noise.
To clarify, are you asking the following: If $x\\sim N(0,\\sigma^2)$ and $y=e^x$ then what is $\\text{var}(y)$ equal to?
@user3447 What is the intended use of this scale? Do you aim at comparing two or more groups (wrt. mean scores) or so?
Necessarily so. There's a smallest possible gap between two floating point numbers. You can't get values between them. The  mathematical notion of closed and open is somewhat less clearly what you intended when it comes to numbers represented on computers.
Hint: SVD. ${ }$
Just for confirmation, I ran the same script in my i3 2120 (3.3ghz) with 4gb rams, 64bits w7: user 0.27, system 0.01, elapsed 0.34.
LOOCV at the end is just a kind of cross-validation. We need a solution for the problem in which we have a small data, we want to select good features and finally have a representative measure to evaluate performance.
@wok, nah. Piece of cake on today's computers. Anyway, you could always dump to a PNG first and that's likely to be good enough for the degree distribution. The OP's graph really isn't all that big.
If you have a lot of data, one viable option could be to randomly split your dataset in half, do exploratory analyses on one of the halves and use those results to form some hypotheses of interest, and then--having **not** yet looked at the second half of the data--do a smaller set of confirmatory analyses on the other half.
$P(N_{11}=3) = P(N_{11}-N_{0}=3) = \\lambda^3 \\frac{e^{-\\lambda}}{3!}$ I guess, where $\\lambda = \\frac{11}{5}$
Perhaps I am misunderstanding the question, but you can't include all levels of a categorical variable into a linear regression because of perfect collinearity. One of your categories will be dropped to provide a base group against which the other groups are compared.
But they also have four different beef products i only have one. Is that what makes the difference?
Perhaps it would help the OP if you could paste the code for your little simulation study?
I think @cohoz is right in that predicting probabilities is a *regression* problem, so you want regression error/performance measures. I'd start looking for literature in the weather forecasting community, e.g. Brier's score (not sure, so only a comment).
I don't get the "every i-th sample" sentence. What is i? Are you just trying to say that every sample is labelled with either with 0 or 1 irrespectively of which data set it is form?
Thank you, these solutions also work great.
Is it **McNemar?**
Many weighted sums of chi distributions won't look at all like gammas. This suggests you are thinking of particular kinds of weighted sums. Can you provide details? Some context?
Can you describe advantages and disadvantages of these methods? A one-line answer is an insufficient one.
(+1) I also have this one in my list, [Power analysis for genome-wide association studies](http://www.biomedcentral.com/1471-2156/8/58).
Do you have just *one* number? Do you have, maybe, an infection count per month? I wonder if this might be modeled better as a time-series, or with a mixed-effects model or the GEE.
Actually, after a little thought, I think you algorithm is ***exactly*** the Sinkhorn-Knopp algorithm with a very minor modification. Let $X$ be your original matrix and let $Y$ be a matrix of the same size such that $Y_{ij} = X_{ij}^2$. Then, your algorithm is equivalent to applying Sinkhorn-Knopp to $Y$, where at the final step you recover your desired form by taking $\\hat{X}_{ij} = \\mathrm{sgn}(X_{ij}) \\sqrt{Y_{ij}}$. Sinkhorn-Knopp is guaranteed to converge except in quite pathological circumstances. Reading up on it should be very helpful.
$\\mathbf{x}_i$ is the $i$th coordinate of the vector $\\mathbf{x}$. The others are simply typos. Sorry about that. I was trying to follow your notation as closely as possible. I would normally use $\\mathbf{X} = (X_i)$ with $X_i$ as the coordinates of the random variable $\\mathbf{X}$. But, I didn't want to (potentially) confuse.
Side tip: tree-based ML algorithms does not require transformation at all and most kernel-based ML codes have pretty clever normalization built-in.
Your new values do have a standard deviation of $1$ because after subtracting the mean from each of your original values you divided by $4.85$. Where does the confusion arise?
What about that simple idea with cluster analysis. Take $|r|$ as the distance (dissimilarity) and do clustering by a selected method (I'd probably choose Ward or average linkage hierarchical). Select the most tight cluster consisting of 10 items.
what is the dimension of said variable?
I'm coming at time series from an applied perspective, so I started as an engineer, then moved into statistics. All the (and by no means has my exploration been comprehensive) books and papers I've read that came at time series from an econometrics point of view have been 'different'. I'm not sure I can describe it any better, but there's always subtle differences in notation or approaches that don't jive with mainstream statistical time series analysis. It's not that one is better than the other, but simply that for best overall coverage, general approaches win.
@RobHyndman, is there any chance that the forecast package will have functions built for transfer functions (fitting and forecasting)?
Did you mean to multiply by (N-1)/N in the code, so that it matches your text?
thank you for your answer. I think I coulden't explain my problem well. Actually I want to put those who answered no to q1, and those who answered to q2 (yes and no) in one new variable. I want to know is it possible or not?
See related: http://stats.stackexchange.com/questions/26252/how-to-determine-the-accuracy-of-a-statement/26335#26335 and http://stats.stackexchange.com/questions/26054/are-observations-consistent-with-each-other It looks like this is a "rater agreement" type question.
What "parametric test" are you anticipating using?  Many (if not most) of them do not require normality of the residuals, but only need behavior that suggests the sampling distributions of appropriate statistics (such as the variance of the residuals) are sufficiently close to the distributions of Normal residuals to allow accurate computation with a Normal approximation.
Yes, the coefficient value has no relationship with the correlation between $x$ and the error terms.
Oh god, thank you SO much !! I have been able to write the command correctly this time and it worked ! Thanks again !
If you have one blocking factor and one other factor, then [Friedman test](http://en.wikipedia.org/wiki/Friedman_test) is the non-parametric alternative to classical 2-way ANOVA with blocks. How did you assess variance heterogeneity? Usually, variance tests are conducted after having accounted for block effect (i.e., tests are performed on raw data adjusted for block means).
Right, measurements are done with about 1/8 second precision -- analyzed in blocks of 1024 samples.  The "event" is a "snore" detected by the sound analysis.  The "interval" values in the file represent the time from the end of one "snore" to the start of the next.  (It would perhaps be more valid to go start-to-start or stop-to-stop, vs stop-to-start, but these are the numbers most readily at hand.)
You should decompose the interaction. You can interpret the significant A effect as "A has an effect *when B=0*". If you want the significance of the A effect for other values of B (which you surely do), then you need to look at simple slopes (i.e. decompose the interaction). You can do this by using the online tool that I link to in my answer.
One thing you can do is take the simulated estimates themselves and compute the actual standard deviation of the estimates.
*You know, the most amazing thing happened to me tonight. I was coming here, on the way to the lecture, and I came in through the parking lot. And you won't believe what happened. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance that I would see that particular one tonight? Amazing!* -- [Richard Feynman](http://en.wikiquote.org/wiki/Richard_Feynman#Six_Easy_Pieces_.281995.29).
I stated in my answer... you compared a model with time only to one with trt, time:trt, and time as a random effect.  You need to make models that test each of these one at a time to have any idea what causes what.  I gave you an example of a model you can compare to tot.mod.  Furthermore, the betas of main effects of models with an interaction don't really mean anything with respect to the question, "does this predictor make a significant contribution?"
Is that a frequency histogram at the top? If you best want to understand your question, try making the normal probability plot yourself, without a computer.
Just because ML is almost always used to estimate coefficients in a logistic regression does not mean it's the only way! We need to bear in mind that the *probability model* for our data and the *statistical procedure* we use to reason about our data with that model are different things. Logistic regression is, fundamentally, a *model*. Thus, Frank's question still stands unanswered: what is not right about the procedure he proposes?
I echo @Scortchi 's comment. Of course, different people have different training, but I took around 10 statistics related courses in graduate school (my PhD is in psychometrics). Each used at least one book.
@user023472 Sorry I don't have the energy today to make an answer about intrinsic discrepancy approach. It is based on the Kullback-Leibler divergence.
Summarizing whether variables agree is answered by concordance correlation. See e.g. http://stats.stackexchange.com/questions/3061/does-the-concordance-correlation-coefficient-make-linearity-or-monotone-assumpti
@SooBin, I would be interested to know how applicable this method still is, having read through it a while ago.
The performance of this approach is incredibly bad and explains why a good deal of published research is non-reproducible.
In "the real world" you don't know how much noise is in the image (or any data), which is why you create artificially noisy images. This is a standard technique in image processing research.
It doesn't matter, they are both tools, neither is useful in and of itself for study. An analogy would be 'should I buy a hammer or a screw driver to learn to be a carpenter'.
It is striking that the majority of the values in this time series occur multiple times (one as many as 11 times). How were these values measured?
In double precision, that's a denormalized value considered equal to zero :-).
Hi Jon, it doesn't seem like to me your question is sufficiently different to warrant a new one. This is borderline on topic for the site, and so it may be better to submit your question to either [stackoverflow or other SAS specific forums](http://meta.stats.stackexchange.com/a/816/1036).
Are these quantitative variables? If so, the simple solution is to use the sum of *cty* to predict for the other observations.
Your first example us the hypergeometric distribution, while the second one is the binomial distribution. This approximation is quite common and works well for large N.
"In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule and its result (the estimate) are distinguished."  (First line of the Wikipedia article http://en.wikipedia.org/wiki/Estimator ).
It is not at all apparent that this method minimizes the loss you cite in a comment to your original question, Davoud. For instance, if "squared difference" is interpreted to mean that a penalty equal to $(x_i-t)^2$ is assigned to each value $x_i$ that is misclassified with the threshold set to $t$, then $t=1$ is the optimal threshold, not $t=0.46$ (which is returned by your code).  What, then, is your *real* question: what exactly are you trying to achieve by your choice of discrimination threshold?
I don't see how the test for B2 concerns itself with testing the constancy of B0 . Consider the following time series 1,2,3,4,5,6,21,23,25,27. Your procedure would not work. Now if the data was 1,2,3,4,5,7,9,11,13,15 then your approach would work since only the slope changed. In my first example both the slope and the intercept changed.
Yes, basically it is a random point count in the gonad, where either 30 or 50 acini where assessed for spawning or not. so my data in each gonad is yes/no/no/yes... 30 or 50 times. From this I calculated the percentage.   Below the first few rows of my data. Each row is one scallop                               density	percent	gonad	temperature
1.455	0	3.1	11.35558
1.455	0	4.7	11.35558
0.59	0	12.44	11.35558
0.59	4	1.72	11.35558
0.59	0	4.28	11.35558
0.59	0	5.39	11.35558
@AriB.Friedman, reminds me of Ed [Tufte's philosophical signs](http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0003pg), [Correlation is not causation, but it sure helps](http://www.edwardtufte.com/bboard/images/0003pp-10887.jpg).
Some sort of plot would be sensible to explore the data.  It is difficult to comment on your other statements without more information such as what "more important" might mean.
Charlie - I'm not sure I understand why you multiply treatXquin2 by 1 for yes. If that is the case then how do I obtain the estimate for the untreated (i.e. treat==0)? It doesn't seem to make sense that I would multiply treatXquin2 by 0. Any help would be much appreciated!
@Glen_b no, I just were talking from my point of view, so these are not my estimates.
For the intuition: Suppose I tell you $|X| = 3$. What can you conclude about the value of $X$, without considering *anything* about probability? Ok, now, what does symmetry buy you?
I think the conventional wisdom is that binning should be avoided--you're throwing away potentially useful information. On the other hand, some phenomena really do have hard edges. For example, 65, 67, and 70.5 are important ages for retirement benefits in the US and it'd be silly to insist that your analysis "rediscovers" those values.
@Glen_b i am practicing on my own .
I would take @gung's comments to the question seriously: the first thing is to do *some* kind of quick fit--even by eye--and begin exploring the distribution of the residuals rather than focusing on the distribution of the responses. I would guess the modes are associated with, or indicate, some attribute of the subjects, so it would be interesting to explore any associations between those modes and all available covariates.
@cardinal, so do you agree that the data in scenario 2 is not identically distributed before losing the gender information?  So we would have a case where they are not identical, but the only way to tell the difference is to use information outside of the variable being looked at (gender in this case).  Yes being IID is a well defined mathematical property, but so is being an integer, can you test whether the data point 3. is an integer stored as a floating point number or a continuous value that has been rounded without outside information about where it came from.
Thanks! I'll go through the problem 26.29.
@JacobMick I've implemented that algorithm / version of Nystrom. Measured in terms of error rate, I've not observed a significant difference.
A related question: http://stats.stackexchange.com/questions/4/assessing-the-significance-of-differences-in-distributions
I don't think my comment is worthy of a separate answer, but since you are doing daily samples, one thing to look for is patterns based on conventions that won't be inherently obvious from the data set. Consider if there is a weekly or monthly pattern, for example, which may be stronger than other intervals. For something like running, you would also need to consider a weather overlay to see if rain affected the outcomes if it occurs outdoors.
@user2238, if you like the answer you may up-vote and mark as correct then ;)
+1 nice question. Unfortunately I don't know enough about it to answer this entirely hence just a comment, but I think the papers by [Meyer and Mittag (2013)](http://home.uchicago.edu/~mittag/papers/BCM.pdf) and [Lewbel (2007)](http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0262.2006.00756.x/abstract) should help with the first point at least.