tttjjj/elt_bench huggingface dataset:
Very cool resource, thank you! They only have two blog posts as of 12/13/2011 but they are both IMO very good and informative.
@user777, my answer at http://stats.stackexchange.com/a/28493 explains and illustrates this phenomenon.  (It even uses PCA to construct the sample dataset.)
Is there any meaning to this matrix representation? Its structure implies, for instance, that all the channel values for a given row of the matrix are somehow related. Your subsequent description of the data, however, suggests not: it indicates you really just have some kind of list of time series and if there is any relationship among them it occurs at common values of *time* (which somehow must be attached, explicitly or implicitly, to each column). It will also be difficult to attack this problem until you provide some details about how a "short" "event" might be identified.
Many thanks for answering my question.
There are indeed various formulas to estimate the population R^2. See for example http://studyforquals.pbworks.com/f/yin.pdf. Fisher's (= Wherry's) "Adjusted R^2" is said to be slightly negatively biased (it is still dependent on sample size while not dependent on number of predictors), so Olkin-Pratt version is probably somewhat better.
@Ama, your are welcome. At this site and similar stackexchange sites, there is a standard way of thanking for the answers. You simply accept the answer. Just press the check box outline to the left of the answer. I am bringing this up only because I noticed that you haven't accepted any of the answers you asked, so maybe this something you do not know.
From [CRAN](http://bit.ly/1jNsSc7): `Factors are currently implemented using an integer array to specify the actual levels and a second array of names that are mapped to the integers.` It's the *coding system* represented here--the mapping of the names to integers--that I am referring to.
Thank you very much for your reply and many thanks for your detailed explanation. As far as I know in a case that we don't have any dependency between the distributions, when we have overlap in error bars, significance test can help us to identify if two distributions are distinguishable or not. In the case of dependent distributions, I am not sure if ρ would play the kind of same role or not. I appreciate if you let me know more that after identifying ρ  what I can do being able to interpret the results of evaluated criteria under uncertainty.
That may work for the physicists, but is sufficiently sketchy that it should raise doubts even in practical applications. Even if you use a really, really "large-enough" number, the resulting "probability" will still not have a finite integral! Either a principled introduction of a true "infinity" is needed or a more rigorous construction of $E$ is required.
You cannot prove the null hypothesis. With sufficiently large N, any difference (no matter how small) will be 'significantly' differentiated from your reference value. It may help you to read my answer here: [Why do statisticians say a non-significant result means "you can't reject the null", as opposed to accepting the null hypothesis?](http://stats.stackexchange.com/a/85914/7290)
@whuber Thanks for pointing out the mistake. I was mixing up $\\mu^{-1}[1-F(t)]$ which is a valid pdf and $G(x)$ which is a valid CDF. I have deleted the incorrect statements. But I do think that for $x < 0$, setting $$G(x)=\\frac{1}{\\mu}\\int_0^x [1-F(t)]dt = -\\int_x^0 [1-0]\\,\\mathrm dt = x$$ as is seemingly implied by the problem is incorrect.
Great! I looked through the contents and a sample chapter and it seems under the application section for each distribution it lists where/how/why was this distribution chosen and to model what phenomena. That's pretty much what I was looking for. I've already gone through the wikipedia links :) I guess I should be able to take it from here - else the forum is always there to help with the clarifications :D
Explaining the documentation to you in this case really amounts to explaining the statistics to you, which is off topic for StackOverflow. R's documentation is never the right tool to educate yourself about statistics. For that you'll need to turn elsewhere.
You mean the Cross-validated values in the plots/table? Well, they are generated by randomly splitting the training data in k = 10 groups for a k-fold CV. You are seeing some of that randomness. Does it change fundamentally how big a tree you need? If so I think you have bigger issues. However, run `set.seed(foo)`, where `foo` is a number, prior to calling `rpart()`.
Yes, thanks for the hint of disjointness.
Just a terminological point: _variables_ are hetero/homoskedastic, not variances. Also, if each $x_i$ has a difference variance, $\\sigma^{2}_i$, you cannot estimate the variances - you would have more parameters (including the mean) than data points.
This issue is usually solved by appyling suitable transformation to the response or, even better, to the parameter of interest (logistic transformation e.g.)
This is a very, very broad question.  Try to narrow it down a little as to what your specific question is.  Perhaps try to explain what you have already tried and where you got stuck...
I can't recommend any automated methods of outlier deletion. I would consider robust methods instead.
As I said to StasK, it _is_ still a point process, though it is [degenerate](http://en.wikipedia.org/wiki/Degenerate_distribution).
It has been introuduced by researchers as a tool for modeling counting processes, please see the link for more information:http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9892.1987.tb00438.x/abstract
$R^2$ is a measure of the amount of variation explained by the combination of all the independent variables.  Dropping one of the independent variables and measuring the change in the $R^2$ would tell you: Variable IV X (that I dropped from the regression equation) explains delta $R^2$ amount of the variation
I used to think like that (that high correlation between variables implies that they carry the same information) until now. By construction and economic meaning, these two variables should be highly correlated. But there is variability of X2 for a given level of X1.
Too little context. How many respondents were there in the survey? How many of them did not fill in the questionnaires? How was the satisfaction question worded? Have you had surveys during previous training cycles? How do you plan to cope with polite insincerity in responses? What is your loss function? Did you check the questionnaire with professional psychologists before conducting the survey? Etc. etc.
Both, I am apparently using the term centering constraint incorrectly.
How many times do you want to run the simulation? I'd imagine R code would get an answer relatively quickly, but can't say without seeing your code.
+1, this is the right answer. For an *illustration* (eg, pictures) of how $f(Y)$ & $f(Y|X)$ can differ, there are some at my answer here: [What if residuals are normally distributed but Y is not?](http://stats.stackexchange.com/questions/12262//33320#33320)
@Charlie an alternative that is often easier to both plot and read is to repeat part of the data and plot a bit more than a complete period, e.g. to plot over 30 hours from midnight to midnight and on to 6am, so the info from midnight to 6am appears twice - much as New Zealand or Alaska appear twice on some maps. I got this idea via Stata guru Nick Cox but I can't find the ref at present. [Tukey (1972)](http://www.edwardtufte.com/tufte/tukey) favoured two complete cycles.
@Josh, You have to define specifically what is `reliable` and `able to be classified as factors`, for both phrases aren't clear at all.
Excellent questions, I asked myself similar questions during a long time. Finally the answer I found was the abandonment of subjective priors. These questions are endless. More seriously, perhaps there are answers more positive than mine but I think that should depend on the professional context.
Check http://blog.okcupid.com/ - somewhere they talk about the underlying matching models.
+1. Stata users can also find a concise review in Baum, C.F. 2008. Modeling proportions. _Stata Journal_ 8: 299-303 which is accessible at http://www.stata-journal.com/sjpdf.html?articlenum=st0147 Even non-Stata users may find this helpful.
Nb., $.25^2 = .0625$. Also, I think you *could* explain the connection to regression methods & how they can help the OP achieve his goals; he isn't asking for a class, so we don't need to only provide hints.
Both are same [see](http://en.wikipedia.org/wiki/Logistic_regression)
I would say if you have a high (or perfect) AUC in-sample, but a very low AUC out of sample, something is probably wrong.  If they're both high, you just have an easy problem to solve.
@RubenReal You're welcome. Glad I could help.
Please, don't just post your homework as is. Tell us what you have done or where you are having difficulties. Homework questions receive a [special treatment](http://meta.stackexchange.com/questions/10811/how-to-ask-and-answer-homework-questions/10812#10812) on this site.
@image_doctor: you should also try with "method=boot632" and  "method=boot", with a high number of repetitions, and see what number those two methods pick.  Also, if my answer is sufficient, feel free to make it as "accepted."
Welcome to CV, @manimal. There's certainly no problem with providing answers to old questions. In fact, if you can contribute something useful to the thread, we would prefer that you do. One note however, is that meta-comments on your answer are unnecessary & best left off. Since you're new here, you should read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info about the site & how to answer questions here.
Thanks for the reply. I recently came across a post using somehow a different formula for this case, e.g. a nested one way anova model as aov(Value~Group/Subject,data). The results are different. Which one is more correct?
Maybe you would like to add key-words relating to the forecast package (like "forecast", "Box-Cox" whatever is available). Prof. Hyndman is very active on CrossValidated. Or you comment in the blog, which he might read. To me, Box-Cox is not only to make data more homoskedastic but also to make it more Gaussian  - which makes it easier to model. In the blog he uses stl which is meant for additive decomposition. In case you have a multiplicative model, the log transform makes it additive - and on generalization of log is BoxCox.
The best approach to estimating an ARMA model is maximum likelihood estimation. To do that, you need to calculate the likelihood of the data given the model and parameters. Any decent ARMA software will do that, but not all of them do it using a state space representation which is needed in order to handle missing values.
+1 Nice pointer to the OR exchange site.
Perhaps the fear of "smoothing" is partly caused by conflating the model "$p(i) \\approx \\alpha \\text{react}(i) + \\beta (\\text{react}(i+1) + \\text{react}(i-1))$" with what would happen if you used "$\\alpha \\text{react}(i) + \\beta (\\text{react}(i+1) + \\text{react}(i-1))$" as a *filter* to estimate $p$. The model describes how you think the probability of pairing actually works - if that model is a reasonable description of the process, it would be expected to describe what happens!  Fitted probabilities will be smoother than the reactivities, no doubt, but it's not a *problem*, it's the model.
Thanks for the reference @user2300643. And thanks for avoiding the financial barrier to access!
+1 very nice answer! You might consider throwing some `sqrt`s in there so that the it's the area of the rect rather than its length and width that represents the data.
I don't understand how to thank you. Please accept my heartfelt thanks for your great help and guidance. Regards - Katherine
Actually, much depends on the value of $p$. For a general answer, see the answers to [this](http://stats.stackexchange.com/a/996/603) question.
My comment applied to the part of the question about trying different transformations to achieve normality.  The OP was trying different transformations to then be able to estimate quantiles.  This is almost futile.
@Clark, I think the prediction of the unobserved response is appropriate here. What makes you say otherwise?
@Glen_b, I got confused because in a prob. dist. the sum of probabilities equal to 1. In this case it will be greater than 1.
Added some latex, see if it says what you want it to. Your second equation seems incorrect; combining it with your first equation it says that $x(t)=x(t)-x(t-1)$, is that really what you mean? Also, from your long run mean I take it that $a$ is an intercept, but then you should probably have an innovation too, right?
(+1) He's also working on a set of tutorials on [Advanced R development](https://github.com/hadley/devtools/wiki), very handy!
When you say 'distribution' do you mean density-function or distribution-function?
Do you know any info on counts (eg total sample size)?  Or is it only proportions?
Thank you.  There are rigorous ways to account for tied groups.  They are important when working with censored (but continuous) data, because frequently the censored values constitute a large tied group.  For the Kruskal-Wallis and Wilcoxon Rank Sum tests, see chapter 18 of R. O. Gilbert, *Statistical Methods for Environmental Pollution Monitoring."  Formulas involving tied data can get complicated, but in some cases (like the K-W test) all you need to do is compute an ANOVA table for the ranks.
This is perhaps the most cynical answer on cross validated. +1
I'm looking for the same thing. I think glmmADMB is what I want, but I can't get it running.
Before you draw any conclusions about which items to keep, try running factor analysis per se, instead of principal component analysis.  In SPSS it uses the same initial menu but a different extraction method (plus you are better off using oblique rotation).  For reasons why, see http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysi .
Although what you say is true, it seems the OP is asking about continuous *target* variables (or classes or response variable or whatever you want to call the thing being predicted).  Your point about splits in decision trees is related to continuous predictors (or attributes or explanatory variables or whatever you want to call the things used to make the prediction).
By the way, is this a duplicate of what you posted here? http://stats.stackexchange.com/questions/920/test-if-probabilities-are-statistically-different
Hi again. I researched and gaussian priors isn't the answer I'm looking for at this particular stage. My question is still regarding the selection of context. Could you see the 'update' section of the question and check if my new selection of context is correct?
Assuming your data are actually paired (are they?), the two tests you list are not the only possibilities. Low p-values suggest a difference.
Thankyou I appreciate your comment, I am personally thinking the mixed-effects model might be better for my data will have a look into how to do this.
Correct, as long as random factors are specified and data are coded correctly, you do not have specify which is a between- or within-subject factor.
James, if you can - please give links to descriptions of these methods.
thank you sir. it was the wrong data set.
@Glen_b what I am trying to achieve is to explain that treatment B is better compared to treatment A for the exact same group of people - based on the measurement M1.
possible duplicate of [When is it ok to remove the intercept in lm()?](http://stats.stackexchange.com/questions/7948/when-is-it-ok-to-remove-the-intercept-in-lm)
you need to give more information. How much data have you collected? What kind of data is it? Are you aiming for a statistical power threshold?
(+1) As usual from you, this is an interesting and well-constructed question.
+1 for Rcmdr. It's nice for when one is just learning R, and offers more of a gateway to a good statistics package than SPSS or Minitab, but the code it writes is needlessly verbose, often. A scatterplot produced in Rcmdr: scatterplot(tab~pct, reg.line=FALSE, smooth=FALSE, spread=FALSE, 
  boxplots=FALSE, span=0.5, data=senate.race), when most of those parameters were defaults to begin with. It writes some very wordy code.
Dumbfounded by this comment. This is priceless and should be kept for posterity: "Actually, the first and foremost reason [for me to ask this  question] was to get some reputation to be able to answer a question on meta".--Felix Dombek, January 31st, 2011.
When i fit the logistics regression on the given data set, only the coefficient is significant. But all the variables are equally important. that's why, i want to check the interaction effect of independent variables with output. what should be the procedure?
I'm not sure I quite understand your question. Is there a reason you don't like or can't do an eigendecomposition? Without loss of generality you can assume all variances are one.
+1, good suggestions (& thanks for the catch).
@whuber Really Really appreciate the patient explanation about the value of missing values.. That is indeed a good tip. I will saw some values with NA and None and ! which I don't understand. Seems like your concern totally makes sense.
+1 The distribution is not likely to be Poisson, though: we would expect sharp differences in accident rates (per unit time) between weekdays, weekends, and holidays, as well as seasonal variation and effects of temporal correlation. E.g., if the average is attained by virtue of a few hundred-car pileups each year, we might expect 300+ accident-free days annually. What can be said is that one very general model supposes the accident rate is a Poisson mixture; Jensen's Inequality then implies $365\\exp(-2)\\approx 49.397$ is a *lower bound* for the answer.
Great answer! I appreciate all the detail on this.
Simply feed the future regressor values to the `newdata` parameter of `predict.lm()`. Make sure they are in a `data.frame` with the same names as used in the formula in `lm()`.
This question makes no sense to me: it looks like it specifies a distribution for a vector-valued random variable, whence its expectation must be a vector, while "np" (whatever it might be) appears to be a number. Could you please clarify your notation and the question?
Do you already have evidence of collinearity in your data set? In particular, do almost all those of a particular `gender` have the same `gender-specific comorbidity`? Also, how many predictor variables do you have, and why do you wish to select only a subset of them? In general, for purposes of prediction, it helps to keep a large number of variables, even those that are not "significant."
The lognormal is in the exponential family - it even says so in the very link you gave! See the second sentence [here](https://en.wikipedia.org/wiki/Exponential_family#Examples), and see [this table](https://en.wikipedia.org/wiki/Exponential_family#Table_of_distributions), right above "Inverse Gaussian", with details of exactly how it is (i.e. $h(x), T(x), A(\\theta)$ are given). This issue is a regular point of contention. If we restrict what we're talking about when we say "exponential family", then the lognormal may be excluded, but the definition given at your link includes it.
You have 3 explanatory variables, so you need those subscripts to be vectors. But for simplicity's sake, imagine you have only 1 variable $X$. $\\mathcal B(n_{x_i},~p_{x_i})$ is the conditional distribution of $Y$ (ie, `obesity`) when $X=x_i$: given that there are $n_{x_i}$ such people & their probability of being obese is $p_{x_i}$, that is the distribution that describes the number of obese people you will see.
With highly (positively) skewed data, a very small number of extremely high values strongly influence the regression.  Until you perform diagnostic analyses of influence in the "significant" regression, its results have to be considered unreliable.
@Vincent At risk of being repetitive, I would refer you again to the TAS article ("Two Pitfalls in Survival Analyses...") because the authors address *precisely* this issue.  They use "multistate modeling" to examine "specific types of survival bias such as *length bias* and *time-dependent bias*."  You should be particularly concerned about length bias, which is a form of sample selection bias.  (What about people who died before they had a chance to ever make it into your study?)  This article, as is typical for TAS, has a good bibliography for further research.
What are sparks?  Are they what I would call spikes?  Spikes are values orders of magnitude higher than other points in the time series and they are isolated.  If you have spikes do you know why they occur.  Are you sure that shrinking them is appropriate?
Correlation finding is a hefty task. Have you seen [this](http://www.powerpivot.com/) tool at work ? It might help you find the proper way of organizing your data.
Do you know that members of the second group aren't included in the first group or are you just willing to mistakenly label those members as belonging to the first group?
I actually didn't do any post hoc testing - I just used an anova to compare the four data sets. The only group that was significantly different than the others was XB. Our hypothesis expected the effect of B to be negated in the presence of A. +1
@gung: right, so you're saying that the approach would be affected by the specific situation. But lets say the measurement errors are independent for the sake of argument.
I meant that the contributions vary in terms of the probabilities. You are right that they are linear in the log odds, but log odds aren't very intuitive (at least, not to me or most people I've talked to).
"*The model that I am expecting is asymptotic.*" -- okay good, that makes sense, but this is why I don't see what you mean by 'log-linear' (and why I wanted it specified) ... and it certainly rules out $f(x)=x$ even on a log-f scale (the log-function *isn't* asymptotic).
A highly readable introduction to this, from an econometric perspective is given in [Rothenberg (1984)](http://econ.ucsb.edu/~doug/245a/Papers/Approximating%20Distributions.pdf).
@PeterFlom Thanks for the information. I will see what I can do about accessing the raw data & update the thread. In the meantime can you point me in the direction of what kind of regression I should look into (any sort of literature)? I am interested in learning more about this in a way where I can apply the concepts learned in excel. My null hypothesis would be "There is no statistically significance increase in csat scores year over year". Thanks again for your insight, I really appreciate your help. I'll update the thread once I'm able to access the raw data.
I think this question can stay on CV (does not need to be migrated to SO), because I think the larger issue is understanding the ideas behind GLMMs & how they relate to the data visualizations that are possible, rather than this question primarily being about the R code required to get this done.
You don't need it: most of GNU coreutils are available on Mac and Windows, and I know an un*x-like environment can be installed on Windows. Otherwise check [sed for Windows](http://gnuwin32.sourceforge.net/packages/sed.htm).
I don't know where the $\\beta\\gamma-\\alpha\\delta$ combination came from, but yes, the approximation (which was not needed for the result but helped motivate it) is $O(\\delta^2).$
No, I have understand your comment. The French Google translation of your comment is already strange, but by combining my skills in English and the strange Google translations, I'm able to get a correct translation. I will be more relax next month, likely.
Jeff Laake from phidotforum noted that I had to reverse c and p arguments to make it work. More after the click: http://www.phidot.org/forum/viewtopic.php?f=21&t=1922