tttjjj/elt_bench huggingface dataset:
:Badgerman . Thanks for the kudos. I should have added that it is possible to actually search for Trend Point Changes in the case that when the Interruption starts the Y series does not simply move to another level but actually begins to trend. Your comments reflect an uncommon awareness of the possible bodels that might be approriate. I prefer computer-based search procedures which "duplicates" the human eye in terms of suggesting the sufficient combination of dummy variables ( Pulse, Level Shift, Seasonal Pulse and of course Local Time Trends in conjunction with any needed ARIMA structure.
A closely related question (with an informative answer) appears at http://stats.stackexchange.com/questions/55692/back-transformation-of-an-mlr-model.  Although it concerns back-transforming a logarithm rather than a square root, most of the answer still applies, *mutatis mutandis.*
Thanks, I guess it solves the second problem. However, the problem of aggregation by different amount of trials remains. Same values of subject means, for example, can be obtained from different number of trials, thus leading to identical p-values.
Can you please explain both? :)  I saw this study where they use FA to group people's personalities. After they derive the groups, it seems they use Multiple Response Analysis and crosstab it with gender (males and females). As if they change the factor scores into zeros and ones (just a hunch). Thanks in advance!
There are versions of correlation more directly intended for such a situation.
As I mentioned in the edit, I don't think this is quite correct because I'm not taking a range about the median. It's what I first reached for too, before being corrected.
Aren't you concerned that a covariance matrix is unlikely to be a useful description of parameter uncertainty for any estimates at or close to the constraints?
While it's a response to a different question, several of the points [in an old answer of mine](http://stats.stackexchange.com/a/25725/2970) can be carried over here. Assad makes some similar remarks in his very nice answer below.
(+1) "Bivariate kriging" is called "cokriging." There's lots of commercial code. For an open source solution `geoRglm` will do it, I believe.
One thing to do would be to apply some form of regularization, such as shrinkage, such as by Lasso or possibly ridge regression. Another alternative - especially if the variables are in some sense related - might be to take the first few principal components and use those in the regression equation.
Had you wanted to use a fixed effects model (generalized least squares with, for example, a continuous time AR1 covariance structure) and if you were a LaTeX user, the R `rms` package allows you to typeset the fitted model in algebraic form with the `latex(fit.object)` function.
I'm not sure to follow: Are you asking for how to make a non-significant effect significant by changing original data? Or is there something that prevents you from running specific contrasts in SPSS when no interaction happens to be significant at the 5% level?
Please avoid abbreviations like "sth". There's no 140 character limit and it's not like typing "something" will break your fingers ... but it will make your post understandable by a wider range of readers.
You might want to investigate the `rbinom` function very carefully. It may take some thought.
@Arun, Sorry if my comments confused matters. It was meant more as a side note. If the two samples are independent, and each comes from a normal distribution with the same variance, but possibly different means, then, yes, a two-sample $t$-test is the natural choice. There are strong theoretical reasons for its use. The interesting thing from a statistical perspective is that these theoretical properties break down in a big way as soon as we let the variances be unequal. In applications, this is often seen. :)
A simple heuristic I thought about using was to just consider elements in bins that represent the upper 3/4 quartiles. In other words, dump the first quartile and keep the rest.
Thanks for your informative response! If you have a link to the proofs, it would be highly appreciated! I am still interested in the actual arguments/proofs/math. :)
Consider an extreme case of correlation.  Suppose all the errors were perfectly positively correlated. In other words, somebody had generated a *single* random number and added it to all the response values.  How certain would you be of (say) the intercept in the regression?  Would you have any clues at all concerning the size of the random value that was added?
Jenks *is* k-means in 1D.
Note that $1-(1-0.05)^3 = 0.14$.
@mbaitoff: In terms of *sampling* from a truncated distribution on a disk, that can be done quite readily by rejection sampling or other methods. If you want the distribution centered at the origin and circularly symmetric, then one only need sample from a single distribution (say, on the *unit* disk) and then rescale appropriately.
What is a "LOESS/ARIMA model"? Do you meant a nonlinear time trend with ARIMA errors where the nonlinear trend is estimated using LOESS?
This is a lot to digest, thanks a lot! I guess it really is a time-series based problem, and I will need a model as a starting point.
Nice idea! -- But could you perhaps explain how to obtain CIs for the median from a life table?
It's impossible to answer this question. Too broad.
This should be on SO with an [r] tag.
Thank you, now I know it means MartinGale Difference sequence
I am guessing is it possible that tm package and Weka use different tokenization algorithms, which results in different vocabulary size. What do you think of this?
How about using some smoother? I guess it depends on what you want to do with the derivative, though...
By the way, I hope you are not comparing a proportional odds model with a multinomial logit model. These models are not nested.
@SamFelix exactly the same package helps you to estimate the parameters of your model (method fit), and as soon as you estimated all the parameters you have the conditional distribution for unobserved variable, so you can estimate it's value with the mean (method predict).
@RussLenth Russ, the goal is to figure out the expected number of intersections at a random sample point (with uncertainty). We have no idea how many rectangles are in the field. We could sample many times to get an average value but a key point that I neglected to mention is that the sampling process is extremely expensive. We do not want to take more samples than are necessary. Therefore, we want to understand the mean and uncertainty (variance or CI) in the number of intersections with increasing samples. We will then make a decision on how many samples are needed. That's the goal.
This is essentially by definition. The filtration is defined with respect to a probability space $(\\Omega, \\mathcal F, \\mathbb P)$. The $X_i$ are (measurable) functions on that space. $S_n$ is then, of course, a function with domain $\\Omega$. That it is measurable is simply an application (via induction) of a general theorem stating that the sum of measurable functions is measurable.
I know Moore is in the car library, but When sample data is provided it is easier for the question asker to understand your response if you use the sample data.
I will use logistic-regression-related methods like Generalized additive models and Kernel Logistic Regression, so these are due to the large sample size nearly not affected by unequal classes.
I don't think there is a book that tells you how many observations you have. Why don't you give an excerpt of your data, then it might become clearer. Or tell us what model you estimate.
This will sound pedantic, but maximum entropy distributions are non-informative whenever you interpret entropy as "a lack of information".  In many situations this interpretation makes sense, but interpretations lie outside the scope of what can be justified with mathematics.
...and a word generates a thousand images. For the benefit of all readers, of all levels of knowledge and understanding, perhaps it would be useful after the picture, to explain how and why it represents examples of type I and type II errors.
It's not entirely clear what the aim is here. Are you trying to estimate a probability from data or to specify a probability model for the process, or something else?
I think we might need more information about the specification for $\\theta$: it seems like user6600 wants it to have a different distribution at different points in time.
If your hypothesis is that the drug affects peoples' heart rates dramatically but in either up or down, you could measure the absolute difference from the first observation and test that.
I would assume those 3 variables were constructed through factor analysis in the first place. So I do not think it would be appropriate to do it again. I do agree with the collinearity
You seem to have a trailing space, i.e., "medium " in some of your observations.
Multiple data points for / observations of what exactly?
Ahh oh, cool! And what's about the case when I derive the kernel from $\\psi$? Did I assume correctly that the regularization matrix should/may be $I$? Or even the same kernel? Or just PSD? :)
I feel like I see $\\mathbb{P}()$ more in the context of measure theoretic probability.
@steffen yea, it is `Business Intelligence Visualization`. I had heard a lot about `BI Visualization` and need to confirm whether there is any difference between visualization and BI Visualization.
What are `a`,`b`,`c` in `J`? How are you randomizing in #1? How are you resampling in #2? Is the randomization/resampling destroying the interesting structure of the data or preserving it?
Depends on the readers of your report & the level of detail you want to go into; but describing the test & giving the observed value of the test statistic & p-value should be fine in general.
What do you mean by an example? Logistically (no pun intended) it is the same as less than six categories.
The Journal of Statistical Software (online) had a recent issue about multiple imputation, and I've been looking at the Big Three multiple Imputation packages for R: `Amelia`, `mi`, and `mice`. The similarities and differences are fascinating. (`Amelia`'s `over impute` is quite interesting.)
Your first question implicitly assumes the two scales are commensurable. This seems like a dubious assumption. Although a statistical test can show that two *numbers* differ, what would that possibly mean in this circumstance? It sounds a little like testing whether the speed of light is greater than the average weight of Florida oranges. Your second question seems less problematic, but it raises concerns about how *self-reported* and *actual* behaviors might fail to correspond.
I don't know; I am not that much of a user of GENMOD.
Bonferroni is a way of adjusting for multiple comparisons, applicable across a wide range of situations, it isn't designed to show which pairwise differences are significant. 

Other than that, this question is very similar to [this one](http://stats.stackexchange.com/questions/16665/how-can-i-get-a-significant-overall-anova-but-no-significant-pairwise-difference)
It is not clear what you mean by "model" and "estimation."  Do you have any data?  Are you trying to use them to fit the upper tail of a distribution? If so, why?--what information do you hope to derive from that fit or what further analyses will you do with it?  What additional subject-matter information might there be about how the upper tail ought to look?
JonnyBoats here is right on.  You can do a lot of tricks with NLP, but analysis of human behavior is the best way to programatically determine quality based on content.  As the saying goes, "Every-time I fire a linguist..."
What kind of demographic data are you trying to forecast? Every data could (and should) be treated differently.
Yes, you're on the right track. Not sure there's much more to say..?
Apparently I know enough that I am beginning to forget some.  Thanks for the reply, in any case!
And here's an additional link: https://www.iser.essex.ac.uk/resources/survival-analysis-with-stata, see especially Jenkins' lesson 6.
(1) Defined where? That seems an odd definition. Why would you specify the distribution of X? You're conditioning on $X=x$. Your confusion seems to stem from that rather strange definition. (2) makes no sense to me at all.
I believe you are unnecessarily narrowing the scope of your question. Why not instead ask *how* to solve your problem rather than asking whether linear regression will solve it?  That will open up the thread to all kinds of classifiers, such as [tag:SVM], [tag:random-forest], etc.
I would read up on LARS and LASSO (via google) and then search for packages with those words in their name (there are many).
@KostasAlex: No problem! Just remember to click on the answered tag if this helped! Cheers.
If there's no meaningful ordering to the data at all, it would seem an odd kind of dependence to check for.
Judging from your question, I suggest you should use already existing implementation.
@IrishStat: Not sure if I can share the data, put a small chunk of it, with a description of what the rest would be doing. It is going to be very similar to the following, just perhaps more accurate. https://www.quantcast.com/p-c1rF4kxgLUzNc . I'm more interested in learning how to work with these sort time series then I am specific results around this particular example.
Thanks! I updated my question a little with new findings.
Thank you, that's what I needed!
"In fact we have recently discussed a paper on CV where PCA" do you have a link to this? I'm very interested :)
*in the first line of equation* is not clear at all. SVM always separates linearly *in feature space*. By using a kernel, such as RBF, the feature space is induced via a nonlinear transformation and as such the linear separation in feature space is *nonlinear in input space*.
This references will be a great help indeed. Thanks a lot!
(+1) Sorry, my post came later and I didn't notice yours. As I think they don't overlap too much, I will leave mine if you don't mind.
I added a follow up question here: http://stats.stackexchange.com/questions/18375/how-to-fit-an-arimax-model-with-r
If I understand your question correctly, you are trying to compare the 3 different groups and test if their responses to the survey are different? You may want to take a look at Kruskal-Wallis test.
It seems to be a common misunderstanding that the assumption of ANOVA is normality of the data, where in fact it is normality of the **residuals** (https://en.wikipedia.org/wiki/Anova#A_model_often_presented_in_textbooks). I would try an ANOVA and report the results here, then compare them with nonparametric results. They hopefully agree a lot.
@PeterFlom Ah I discovered I have seriously confused myself! Updated the question with the correct plots without squaring the residuals. Seems like the r^2 became lower for both the training and testing set after doing the log transform. I'm guessing the log transform does not really help, in this case is there another transformation that I can use? Can you deduce anything from the shape of the quantile normal plot?
You can use the HMM package: http://cran.r-project.org/web/packages/HMM/index.html.
I don't think that is possible. Could you tell us more about your problem? If you are trying to predict "churn" then the data *are* labeled (by churn).
Are you sure you are picking the scores of the 13 PC's associated with non-zero eigenvalues? These should be orthogonal by construction. One check: if you got the eigenvalues from the correlation matrix, are you centering and standardizing the data when you compute the PC's scores?
I'd second @gung's mention of the binomial (/bernoulli) model; there are a variety of tools for modelling such responses over time. If you're trying to model changes of state over time, you might want to look at things like Markov chains as well (those aren't competing suggestions, but complementary ones, dealing with different aspects of the problem).
I think this question is subtly on-topic. I don't think the real issue is 'what R function...' or 'how do I use the R function...'. The problem seems to be a confusion about what t-distributions *are* & how they are similar to / different from normal distributions. In addition, "how would a non central vs a central t affects [sic] my sampling?" is clearly on-topic.
@B_Miner Precisely.
Suppose a variable has three levels, low, med, high. An ordinal analysis could suggest no difference between low & med, but significance for high. The parameter estimate could provide information such as 'when variable X is high, the effect is estimated to be 2.5 times greater than low or medium' - hence direction & magnitude.
In response to your flag, one way to attract attention on your post is to award a bounty on it. I don't think it is necessary in this particular case. On the contrary, I think you should give more information on your particular study: The first example you provided has to do with reinforcement learning while the second is about validity of a measurement scale (and I don't really see how 'relative' applies in such circumstance). What are you interested in, and what application do you have in mind?
This is where repeated measures MANOVA would be good.
Ah, figured it out. $p_i(\\theta) = \\frac{1}{1 + exp(a_i(b_i - \\theta_p))}$.
If you do regression on just the three points in the bottom table you have junk. That's a way overfitted model that doesn't account for the variation in the data.  That's not allowed.
+1, excellent discussion of the issues surrounding suppression of the intercept. I once made a similar point ([when is it OK to remove the intercept](http://stats.stackexchange.com/a/32509/7290)), but not nearly as well as this.
You may want to look at 1932, the interaction of G and P cause it to be a high leverage value, potentially inflating the significance of interaction term in the model `VP ~ G*P`.
@ocram, I've already understand how it comes. But still a question: in my post, the standard error has $(n-2)$, where according to your answer, it doesn't, why?
Thank you! I guess I'll just use your approach for now.
If you don't mind, [this](http://i.imgur.com/fFp27wE.png?1) is what my eventual boundary looks like when plotted in R. Does this look correct given a quick glance?
@PeterFlom I think OP is referring to standardised t-scores as [discussed here](http://stats.stackexchange.com/questions/13412/what-are-the-primary-differences-between-z-scores-and-t-scores-and-are-they-bot). It is common psychology terminology, but conflicts with statistics terminology.
I'm still confused about the p-value. The very small value obtained (`<2e-16`) is for which null hypothesis? `H0: a=0` or `H0: a=0.0062095`? What I wanted was the p-value for `H0: a=0`.
You should at least mention whether order is a part of your 'similarity' score.
In addition, I warmly recommend the [vcd](http://cran.r-project.org/web/packages/vcd/) package, but see the accompanying vignette: [Working with categorical data with R and the vcd and vcdExtra packages](http://cran.r-project.org/web/packages/vcdExtra/vignettes/vcd-tutorial.pdf).
+1 Here is a [link](http://en.wikipedia.org/wiki/Seemingly_unrelated_regressions) for info about seemingly unrelated regression.
The formulas for the univariate folded normal are known. The same formulas can be applied for the variance of $|x_i|$, since $x_i$ is normal. I am stuck at computing $E|x_i x_j|$.
@It seems to work with the dyn function, rather than the dynlm function.
I don't know why anybody downvoted this question, but it may be related to the ease with which Googling `"average partial effects"` (or, better yet, `"average partial effects" definition`) turns up great references.  Nevertheless, a clear answer by an expert would be very welcome here.
I was assuming the data was made up as an illustration.  We all die, smokers & non-smokers alike.  Real studies on smoking & disease have dozens of covariates at least. [Survival analysis](http://en.wikipedia.org/wiki/Survival_analysis) is used when you're measuring time till death or till the end of the study. If for some reason you were conducting an experiment where you made rats smoke 100's of cigarettes a day & that's all the data you recorded over its course, logistic regression is still your best bet.
@JoelW. Analysing data that has already been collected.
The formula for $R^2$ is different when you have no intercept. You *cannot* compare them.
I suspect this is because where you see two factors, one the subset of the other, R sees two unrelated factors.  Just a hunch: try starting a fresh R session, typing `options(stringsAsFactors = FALSE)`, and then running your code.  That would prevent your original `test.pred` from having its own factors.
Good point. Perhaps a reciprocal transform would help, but I have no experience with it.
Can you explain this further and give references for those of us not so familiar with the likelihood paradigm.
@Glen_b See my edits to the answer for examples. I would say that they at least have to overlap at some point, or have some sort of temporal pattern. If I have a time series of light measured every hour, and a time series of temperature measured every 5 minutes, I can get a reasonable lag estimate out of the ccf (given a long enough time series). But others may know better than me.
@Shane. Fair enough. I used to use SPSS. Now I use R. R revolutionised the way that I think about and conduct data analysis. It made data analysis fun. I can't speak too much about Stata and SAS, so I'll leave it for others to justify why they might be revolutionary.
I have revisited this problem again, and have switched to R and am using Zuur and Ieno as a guidance. Still many issues, but in general I think by using varIdent my model diagnostics look like they have 'minor heterogeneity'.  Plotting residuals against fitted looks good, resids against each covariate provides some funky results for one of my model variables (elevation) - mostly a function of small sample size at high elevation.  Thanks for  your comment on fitdistrplus.  Now that I'm using R and Rstudio (love it!) this will be handy!
It depends on what journals you read :-).  The Dutch in particular have been doing a lot with applications of spatial statistics and geostatistics to soils, publishing in *Geoderma* and other places.  BTW, check out [Is R^2 useful or dangerous?](http://stats.stackexchange.com/questions/13314/is-r2-useful-or-dangerous).
The issue with this cheatsheet is it only concerns categorical / ordinal / interval variables. What I'm looking for is a method allowing me to use both numerical and categorical independant variables.
@alto: I think that should read "there are 2 possible responses we can record for a user".  Does that make more sense?
@mpiktas: Edit my question in response to your comment.
Did you see my reply at http://stats.stackexchange.com/q/14483?  It explains why the shapes must vary when the transformation from $x$ to $y$ is nonlinear.
We need more information than that: What are your variables? Is it a single time series? Multiple sites? What have you done so far? Can you share (a snapshot of) the data? What is it you need to check about the submersion time - the submersion of a particular area or overall, the mean submersion time or something else?
'Prior Predictive Distribution' is a helpful name, I see how it is an expectation about what the data distribution might be, before seeing any. I used this interpretation in an answer above, but I had a doubt. I equated calculating $P(Data|S)$ to actually choosing a model from the gamut of $Params$, i.e. what actually happens is that $P(Params)$ changes to $P(Params|S)$. But suppose I calculate $P(Data|S)$ as a KDE. Then I could interpret this in 2 ways: a. I have chosen the $Params$ configuration for which $P(Data|Params)$ looks like the calculated $P(Data|S)$
At present, this is really more of a comment. If you want to keep this as an answer, would you mind expanding on it a little bit?