tttjjj/elt_bench huggingface dataset:
I love you man. Thank you very much for all the hard work it took. You were the only one who understood my real question. I found the code I posted last week but I was having trouble with parameters. I'm a little bit worried about the fact that I have 40 fits to make for each experiment but honestly I think it's the best way to analyse these data. Thank you again!
Thanks for your help, but why the first row of G matrix should be (-1,0, 0). I think it should be (1, 0, 0). Am i right?
(1) Redo the math. (2) Entropy can be measured using any base of logarithm you like, as long as you are consistent. Natural, common, and base-2 logs are all conventional. (3) It's really a mean discrepancy between the distributions and their average. If you think of each distribution as a point, they form a cloud. You're looking at the average "distance" between the center of the cloud and its points, kind of like a mean radius. Intuitively, it measures the cloud's size.
OK, then in order to answer this question we would need your definition of "uncertainty" or "error bar": how are these calculated and what are they intended to represent? (Common choices include one standard deviation, one standard error, and two standard errors, but many others are possible.) Also, the answer depends on how the "best-fit" line is estimated: if it uses weighted least squares (as strongly suggested by the availability of uncertainty estimates) the answer could differ from an unweighted least squares solution.
Posted a followup question http://stats.stackexchange.com/questions/104277/cluster-analysis-over-time-series-samples
I think supplying some more information about the problem you are considering would be helpful. Otherwise, the only really authoritative answer to either of your two question is: "*It depends*", which I imagine you will find unsatisfying. :-)
I was trying to avoid using R because I didn't know how but now I do so I can do that now!
@Marco: just bear in mind that I have not read the book, so I may be wrong! :)
One batch would be less than $10^5$ values.  A year's worth (25 batches) would still be only a few million values, requiring perhaps 10 MB uncompressed storage. Is there some kind of restriction on your storage or computing systems that precludes storing this many numbers at once?  For systems with such restrictions, there is no problem updating the moments: the challenge lies in updating the quantiles. Both problems have been addressed in threads here.  Try [searching for "online"](http://stats.stackexchange.com/search?tab=votes&q=online%20quantile).
Who suggested that human performance is normally distributed?! The 80-20 principle was suggested by Pareto (1906!).
How equal do you want them to be?  The range of values for different was infinite.  Are you proposing to assess the probability of exact equality?
See http://stats.stackexchange.com/questions/2684/how-does-correlation-among-repeated-measures-work-for-repeated-measures-power
Thank you for pointing me in the right direction! I had thought the binomial distribution would be the right one to pick but I was wondering whether I ought to be looking at a confidence interval for a population proportion based on a normal distribution. I shall go and refresh myself in the right chapters of my old school textbook! thanks again.
@user423805 I have changed it to read statistically significantly different from 0.  Formally that means that you test the null hypothesis that the correlation is zero vs the alternative  that it is not 0.  Then compute the two-sided p-value for the test statistic.  Generally statistical significance mean p-value <=0.05.  Sometimes other values are used to define statistical significance (0.01 for example).  Most time series software packages that include mutiple time series can do these tests for you.  Our friend IrishStat can speak to this regarding Autobox.
I think it is biased, like any correlation. I'm not sure for the second point, but to hint: knowing that $r_{x(y.z)}=\\sqrt{R_{x.yz}^2-r_{xz}^2}$ if you take for those R and r their unbiased (adjusted) versions, won't the result be also unbiased?
The questioner said logistic regression... assuming it was ordinal logistic regression then that is the appropriate method.
The main thing it will do is provide measures of accuracy, that (in principle) take account of all sources of error.  ML typically just gives you a point estimate, but no measure of how accurate you expect that particular estimate to be.
Thanks a lot for the suggestion! The two $Q$ values presumably follow chi-square distribution under the random-effects model framework of meta analysis; however, their independence is indeed something that can't be reasonably assumed.
For those interested, here is an [online p-value calculator that takes r and n](http://www.danielsoper.com/statcalc3/calc.aspx?id=44).
possible duplicate of [How to calculate the p-value of parameters for ARIMA model in R?](http://stats.stackexchange.com/questions/8868/how-to-calculate-the-p-value-of-parameters-for-arima-model-in-r)
I was talking about the Tukey's Studentized range test. I will post another answer.
I pretty sure I don't follow your comments on my answer tbh.  But all the talk of indicators suggests that you've missed the crux of Bayesian inference: you never actually *choose* parameters. In particular, you're not searching for the best ones, or the best range of them, according to some pre-specified criteria. All you do is *move probability around over unknowns* (here that means over the possible values of the parameters) as new observations come in.  The customary notation elides this (confusing $D$ and $d$ for example) which does not make things easier.
That's a lot of interactions there.  Try making it a bit more additive, variable by variable, starting with say `mass ~ leaf.species*day + cond.time  ...` (if that makes sense).  That process should both pin down the source of the numerical difficulty (though your analysis sounds right) and indirectly give you a better idea about the substantive implications of the analyses possible given your design.
I want to write $Y_t$ as : $Y_t-aY_{t-1}=w_t+bw_{t-1}$ where $w_t$ is white noise and $a$,$b$ constant. 

I compute $Y_t-4/5Y_{t-1}$.
I get : $Y_t-4/5Y_{t-1}=u_t+v_t-4/5v_{t-1}$ but I cant write it as $w_t+bw_{t-1}$. Any help
I understand local linear regressions, thank you. What I do not understand--and I suspect few others understand either, given there are no responses to this question so far--is *your particular terminology*. A local linear regression is used as a smoother (or "estimator" of a curve); thus, it does not (in any natural sense) have any "intercepts" at all, nor does it have any "slopes": it fits values to a set of specified independent values. My first comment represented my initial guess concerning what you might possibly mean; evidently I did not read your mind correctly. I still cannot.
Thanks! for giving the starting point for my analysis. I am little biased towards Bayesian philosophy, any inputs in this regard is highly appreciated.
Does the correct answer to this one include knowing that there is a controversy about whether fixed marginals make sense and having an informed opinion on the subject?
If the package is able to fit an HMM then has to be computing the probability of sequences given a model, so it's got to be in there somewhere. And the Viterbi algorithm is not what you want; that computes the most likely state path. All you need is the probability of a sequence given a model, which only requires some multiplication and summation.
Thank you so much for the suggestions. I will conduct and report both tests to be on the safe side.
Fantastic answer, thanks for the effort you put into it.
OMG: With sample size so small, the power will be low.
@gung I don't believe [nptrend](http://www.stata.com/manuals13/rnptrend.pdf#page=3) requires panel data. You just need the groups to be ordered.
same reason as here: http://stats.stackexchange.com/questions/76869/is-there-an-implementation-of-item-item-collaborative-filtering-in-r
Hi there, can you explain what event you wish to simulate? Is it bus arrivals at each bus station?
Apparently this is nowadays called a "quilt plot" http://www.plosone.org/article/info:doi/10.1371/journal.pone.0085047
Two comments: (i) In general it is not possible to reparameterise any asymmetric s-shaped curve in terms of this three points due to identifiability (*e.g.* for a one-parameter curve, these three points are a function of it); (ii) There may be many cases in which it is possible but these points cannot be found in closed-form, making the reparameterisation impractical.
You run into trouble at the very first line, because it does not correctly characterize a confidence interval. Please check out [this related thread](http://stats.stackexchange.com/questions/11609) before continuing.
Err, actually you can only do ML estimation on objective functions that are also likelihoods.  That's why there's an 'L'.  This rules out quite a lot of functions.  What you mean, I think, is that you can minimize any parameterized function you like.  Indeed you can.
Thanks, that's great. The paper also stated that the first set of post-scan questionnaires (5 mins after) for the 97 particpants who did return the second set of questionnaires were compared to those of the dropouts. No mention was made of any indication of representativeness. The paper then asks WHY they did this- I'm assuming that is to check that scores weren't significantly higher for the dropouts, as particularly high anxiety could be the reason for dropping out? Wow, talk about a trick question!
I answered this question (perhaps in more detail than you care for!) at http://stats.stackexchange.com/questions/16921/how-to-understand-degrees-of-freedom/17148#17148. To summarize, for the $\\chi^2$ distribution to be valid you *must* use MLE and you must base the MLE on the counts rather than the raw data, as well as use sufficient amounts of data.  The lowering of the test statistic is compensated for by reducing the degrees of freedom.
I do not know why I divided by 2 either. These things happen. Fixed the mistake in the original answer.
Conversely: MCMC could teach one to write faster code and to learn from the pain of waiting for simulations to complete.  This has been my experience with modeling: if it takes a long time to run, I may benefit from learning how to make the code faster.
I don't think the discreteness would matter unless you had an awful lot more than 20 data points.
Thanks very much for your response.  You're right: I was thinking of ratio when I said scale.  This is not for detailed statistical work, but I was interested in the intuition of whether one effect size could be thought to be a scale factor of another comparable effect size.
@MichaelChernick underreporting is generally a problem but if you assume that it is rather constant over time then looking for shifts in trends/rates of reporting would not be affected by underreporting.
That helps *a lot*. The 0.00001 is because you have exact 0's? Is there a spike of values at 0 or is this just rounding? Note that `fitdistr` is already giving you parameter estimates, that's where you'll want to start. A test based on the likelihood ratio may require also combining the two data sets and fitting that. Alternatively, with large samples you might base a test off treating the estimates as asymptotically bivariate normal, but this would require having an estimate of the covariance of the two parameter estimates for each fit, not just their standard error.
This problem is now degenerating into sheer nonsense.
Great answer! Especially the three last paragraphs are very useful. How would you extend this to describe the continuous case?
It's a bit tricky to say there's an error when we don't know what $<.>$ means (context might help), but indeed it must be the case that since $\\bar x$ is unbiased for $1/\\lambda$ that $1/\\bar x$ will be biased; it's a consequence of Jensen's inequality. This is not necessarily a problem, most MLEs are biased; it's not clear to me that unbiasedness is especially important (generally I'd tend to think something more like low MSE would be more important than unbiasedness).
Closely related: http://stats.stackexchange.com/questions/30728/how-small-a-quantity-should-be-added-to-x-to-avoid-taking-the-log-of-zero, http://stats.stackexchange.com/questions/41361/choosing-c-such-that-logx-c-would-remove-skew-from-the-population, and http://stats.stackexchange.com/questions/298/in-linear-regression-when-is-it-appropriate-to-use-the-log-of-an-independent-va.
Do you mean forecast::forecast.Arima()?
The popularity of expressions like "correlation does not mean causation" may lie in affection for alliteration and assonance. Why pick on correlation? It's no worse than many other methods. There are shelves full of books on causation, so the question can't be decided briefly, but I'd venture riskily that causation is mostly outside statistics. Nevertheless statistics can throw some light on causation, if only by underlining which relationships are broadly consistent with a causal idea, and which not so. That does usually work better with a model, which correlation doesn't provide.
Whether or not you use `R`, it may benefit you to read my answer here: [Alternatives to one-way ANOVA for heteroscedastic data](http://stats.stackexchange.com/a/91881/7290), which discusses some of these issues.
The closest distribution to your data is the empirical distribution... Which can be good enough, depending on what you plan to do with it. Tell us more. If you want a "classical" distribution, you have to first decide that you want eg an exponential, then you can fit it to your data.
A similar question was asked afterward http://stats.stackexchange.com/questions/8196/why-is-the-null-hypothesis-always-a-point-value-rather-than-a-range-in-hypothesis/8198#8198
Can you make your problem clearer? What does $x$ and $y$ refer to?
There is `rollapply` in package zoo, which could be used to apply `acf` (or your own custom made function, but take care with the de-meaning that `acf` does by default) in a moving window.
Thanks for the detailed answer @user11852, really appreciate it!
Thanks for the pointers, @amoeba.  I'll check them out.  I'd upvote your comment but I don't have enough reputation. :-)
How do I follow up such a significant interaction between categorical and continuous variable? That is, how can I breakdown the interaction effect for each sub-category of the factor variable?
What is the basis of the pairing, Michael?
Added more information about my data to my initial inquiry!
I can't figure out what that last paragraph is supposed to mean...
In the earlier question, he pointed out that the goal was to compare the SD of raw data with the SD of smoothed data with the SD of highly smoothed data. I don't think the F test (or Bartlett's or Levene's) tests will do any more than complete the circle. The whole point of smoothing is to reduce the variation, so tests that compare variation will tell you smoothed data are smoother. I don't see how this circular logic will get you anywhere.
Very informative and useful answer. The focus on R doesn't really detract from its value since you documented the statistical logic,
Thanks a lot, I will these models out.
@Zach : Marginal effects are just a linear model estimated on top of a non-linear model. If you want to report the results of a linear model then the best way to get those is to directly estimate a linear model and cut out the middle man. So to me marginal effects are poor summaries of a linear model; poor in the sense of hiding all the problems associated with linear models rather than solving them.
@Harrell: Thanks for your comment. Here, I want to give as much importance to sensitivity as to specificity.
+1 for pointing out (in the last paragraph) the assumption of homoscedasticity.
Thank you for your answer. It was very helpful. I use the following code:                                               rows = c(1:2346)
bootDist = c()  
for(i in 1:1000) {        
  bootRows = sample(rows, size=435, replace=F) 
  bootDist[i] = cor(tjornres[bootRows,1], tjornres[bootRows,2])
}
summary(bootDist)                                     hist(bootDist,)                                                         I hope this script will help other student.
Thanks for your answer. Now I have things clear and already got in touch with the authors
Your imagined autocorrelation function does not sound right. Please edit your question to include some _definitions_ of terms. What exactly is meant by a Poisson process? Is $N(t)$ the number of arrivals in $(0,t]$? Is your definition of autocorrelation function $R(t_1,t_2) = E[N(t_1)N(t_2)]$? If not, what is your definition? After all that is agreed upon and settled, we can begin to address your concerns about a _non-homogenous_ process.
I tend to think that writing something from scratch when a suitable library is available is almost always a bad idea. All arguments you could invoke against open-source software (potential for bugs, no guarantees they were properly tested, not necessarily developed by statisticians/academics/economists/software engineers…) are even more valid against home-cooked solutions.
Michael, one other thing: you write that "To assign 0 weight [to] any of the possible assignments could create bias," but this doesn't seem right.  It seems possible to systematically exclude certain vectors of assignments in a way that preserves the possibility of unbiased estimation.  I've given an example at http://stats.stackexchange.com/questions/37775/random-assignment-why-bother.  Of course, it's possible that I am overlooking some important flaw in my example.
See also: [How to generate correlated random numbers (given means, variances and degree of correlation)?](http://stats.stackexchange.com/questions/38856/how-to-generate-correlated-random-numbers-given-means-variances-and-degree-of)
To answer your last comment: The one-sample t-test can also be viewed as a special case of regression. The model simply consists of the intercept (=mean) and the error term, i.e. the response is a shifted error. Since shifts are irrelevant for any assumption, it's equivalent to talk about data or residuals.
There are hundreds of questions on this site about similar data. Did you look at any of them? Search [likert] and see http://stats.stackexchange.com/questions/31598/is-amazons-average-rating-misleading for one example.
The impact of format is a non-issue now. We would like to merge test/confidence responses from other institutions with our own.  Is there a quick way of checking the stability of our test/confidence responses?  Ideally, students with a higher confidence score associated with each question should have a higher total test scores. We are unsure of whether to do this case/case or item/item (aggregate confidence vs. test scores).We'd like to check the correlations pretty close to 1, we will merge institution wide data.  Is there a more efficient way of checking ratings/test accuracy correlations?
OK.  Maybe it would help clarify your problem if you explain why logistic regression is not ideal here.
I think you're wrong about Cramer's $V$ requiring equal expected frequencies...Do you have a source to support your claim? I can't find any myself, and the formula certainly doesn't require equal frequencies to produce a result.
Without any more information than this, how do you suppose anyone would be able to answer? You tell us nothing about the context or your purpose and both are essential for providing objective, good advice.
(ctd) ... and you'll see that in a residual plot.
+1 Good question. I've never really used decision trees but it seems like method 1) will be much more aggressive, which may be desirable for extremely large trees, but method 2) will possibly preserve some subtleties that method 1) loses. Does it just come down to a trade-off between overall accuracy and computational efficiency?
(+1) This is great.  The only thing I'd add is that the $F$ value is the same as $t^2$ for the slope (which is why the p values are the same).  This - of course - isn't true with multiple explanatory variables.
It's wise to be cautious about issuing extreme or universal statements, like something is "always false," because such statements are usually (always? :-) incorrect. In this case, you seem implicitly to assume that $H_0$ is simple. Composite null hypotheses frequently are true and have abundant, convincing evidence in support.  But how has the conversation turned into whether the null is true, when the question concerns when it is *false*?
I think one logical piece of this explanation may be missing: how exactly does one use indicators to obtain *sample* medians? I can see how when $x$ is the *underlying* median, the indicator $X_i\\le x$ will work: but this indicator does *not* coincide with the sample median or any function of it.
I am pretty sure the minimum of that derivative is $\\frac{30}{e^{11}}<0$, at $x=e^{11}$, but maybe Will's calculus is better that mine.
Combinations of I(1) processes can be both I(1) and I(0). If the a linear combination of two I(1) processes are I(0) the two processes are considered to cointegrated and there is a long term relationship between them. If there is no relationship between them, the combination of the two I(1) processes would be I(1).
The other issue will be compositional effects due to sampling changes from one year to the other. I imagine scale usage heterogeneity and sample compositional effects will swamp any other  factors.
Before you registered your account on this site, the question appears to be officially owned by "nobody" (the box containing asker's information is grayed out). As soon as you registered your account and linked it to your MSE account, the software should, and did, automatically link this question back to your spanking new account on this site. So there's nothing more you need to do.
All contributions are warmly welcome, but those without any explanation will be deleted.
Welcome to the site, @DanielEzraJohnson. Would you mind explicating your answer a little bit? We'd like to know a little bit more about how you're thinking about the problem & why you think this is the appropriate answer. Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info on CV.
If you have a model $y = \\mu + \\xi +\\eta$ where $\\xi$ is a zero-mean error term representing underlying variation about the population mean and $\\eta$ is a zero-mean error term representing measurement error, what's the difference from a model where $y = \\mu + \\epsilon$ where $\\epsilon$ is the variation in the observations around the population mean? (i.e. where $\\epsilon =  \\xi +\\eta$)
(+1), however I think one has to dinguish between the goal of ranking (measure:AUC) and separating the two classes (measure: Accuracy). In the former case, given a probabilistic classfier like Naive Bayes, imbalance playes a lesser role, I suppose. Or should one be worried in this case, too ? Another question: What do you mean by "post-process the output" ? Converting scores to actual probabilities ?
I don't have an answer offhand, but chapter 18 of _Elements of Statistical Learning_ is devoted to this topic and covers, I think, all of the techniques you mention.
Thanks a ton, Gerko! I was trying to figure this out yesterday, but was unable to fix it. Now, this should probably be migrated to StackOverflow.
Simulation supports your answer, and even with a transpose operation added to handle cases with more items than subjects, sapply beats diag(cor()), but only when there are notable asymmetries between the N of items and the N of subjects.
Thanks again, Jeromy.

The data will be used for GLM mixed effect in R.

I think your last solution is probably the best, although I am not convinced that Z-scores should be computed across all ratings. 

If the z-scores represent a value against a particular population, shouldn't I split it by group, given that one group (eg adults) may perform differently than another (eg young children), and therefore represent a different population?