tttjjj/elt_bench huggingface dataset:
But normality is *never* satisfied. It's sometimes a reasonable description of the data, but they're not actually normal.While it's sensible to check for non-normality when you assume it, it's not particularly useful to test it (for the reasons I described above). I do a qq-plot, for example, but a hypothesis test answers the wrong question in this situation. t-tests and anova usually work reasonably well if the distributions aren't heavily skew. A better approach might be to use procedures that don't assume normality - perhaps resampling techniques.
Yes, except under certain assumptions. In a linear regression model, this is the assumption of linearity, in a Cox model, proportion hazards. With linear regression, you can make inference about correlation and first-order trend in curvilinear bivariate relationships. Removing high influence/high leverage points *changes the scientific question significantly* and interpreting results as they would apply to untrimmed parameters, an "unconditional population", is wrong.
+1 to all of this. I'd argue that the best "test" (meaning check) of normality is graphical, through a normal probability plot. If a dataset is not normal, it helps you see why much more clearly than any significance test. Also, as is common, if a dataset is declared not normal, it may help you see that the non-normality is nevertheless mild, although such judgement comes partly from experience. In either case, learning that a distribution is not normal is only a first step: the challenge is then to say what kind(s) of distribution are a better approximation.
The World Bank is [going the extra mile](http://data.worldbank.org/news/accessing-world-bank-open-data-in-stata) with open data and maps, for Stata and R.
You can write in LaTeX on this site, just put the code in `$ $`.
@Henrik try lavaan https://github.com/yrosseel/lavaan which is currently the most comprehensive SEM package for R. It allows you to fix/equate coefficients so you can also incorporate the higher order factor. The syntax is also very intuitive.
I suggest you find someone localy who can help you face to face.
@Mike Yes, your comment is really welcome and this was what I tried to point to in my last sentence (optimize FN/FP rates depending on the context). Thanks for noticing the typo (which was already addressed in a preceding post...)!
Thank you. I figured it out and used the spearman command. I am just seeing these responses now.
@kjetil, I first saw of it in Wikipedia when studing the Gauss Markov theorem which asses that least squares is BLUE. I have used on my thesis and nobody complained. It makes sense because for any fixed $\\alpha$, Var[$\\alpha^t X$]$\\geq$Var[$\\alpha^t Y$].
@Anthony Here is a source provided by Rob Hyndman on what is available in R on time series analysis.  It includes what is in CRAN and covers a wide variety of methods including robust filtering.  http://cran.r-project.org/web/views/TimeSeries.html
Thanks @Vincent. From this it follows that `importance(rf)` is equivalent to `rf$importance/rf$importanceSD`
Thanks! I searched for community detection and this was extremely helpful: http://perso.crans.org/aynaud/communities/
I have recently become a big fan of lavaan. See my update to the question above. I really hope development of lavaan continues!
+1 This is a good answer to a nearly unanswerable question. But I'm pretty sure you meant to open by stating that *unfamiliarity* with `R` is an impediment, not familiarity with it!
At a different level, you should understand that because there are no solutions to the FONC and $f$ is smooth, then there are no minimizers or maximizers.
Note that the p-value on a subset and a p-value of the whole data are *not* two different estimates of the same thing! p-value is a function of sample size. Might there be a way that the computational burden could be reduced? Failing that, one might combine p-values (or even in some cases compute a combined test statistic) from mutually exclusive subsets.
In linear regression coefficients can be interpreted as derivatives with units of measurement in numerator and denominator and that can be helpful. With ordinal logit, people don't usually even try, but just look at $z$ and $P$-values. Hence rescalings seem a waste of effort.
Thank you: I had misunderstood this table. It shows "assumed cross-correlations for the cost streams of existing generation assets."  In order for your question to be answerable, you will also need to assume correlations among the different cost streams (within each technology).
ANOVA *is* meant for cases where you have a continuous outcome and categorical/nominal predictor(s). What would take ANOVA out of the running would be a failure to meet other assumptions, as @Mark has said....But maybe you want to treat minutes as a continuous variable too?  You could generate a scatterplot of % (Y) by minutes (X).  You could even run a regression, again if assumptions are met.
Did I understand you correct that you have 2 within-subject factors: 1) context, which differs by some experimental condition (e.g. indoor experiment vs outdoor experiment), 2) trial, which is _simply_ a repetition, an attempt, of experiment. And you'd like to do a PCA in each of the conditions, but it stops you that you've done not one but several trials of the experiment.
You might find [Ben Bolker's notes](http://glmm.wdfiles.com/local--files/examples/overdispersion.pdf) useful here.
If you think it is the case, feel free to put this off.
As you have pointed out, I'm mostly interested in Q2 as Q1 has been addressed using a different set of measures elsewhere and also all evidence and literature points to that. Unfortunately I don't have a huge sample so I guess regarding Q2, I'm going to focus mostly on identifying those measures that can show the decline rather than ranking them.
PDE can be taught in many different flavors.  One might concern itself with explicitly solving basic examples, and another might be concerned with general existence and uniqueness of classes of problems, and yet another might be on numerical methods and when they work properly.
(+1) This is nice. But, note that it depends on the assumption that $\\mathbf X = (X_1,\\ldots,X_n)$ for $n \\geq 2$ is a *random sample*, so we should probably state that explicitly (**Edit**: You did already.). In particular, if we have something like, say, certain generalized linear models (e.g., logistic), then it can turn out that $\\mathbf X$ itself is minimal sufficient.
Usually in VAR modelling the effect of one variable to another is measured using impulse response functions. Do not forget that your model is endogenous so depending on the relationships with other variables the differing signs in two different periods do not necessarily means that there is differing relationships in those periods.
It's not weird at all. You can compare floods (rare) with other events, cases of unusual diseases with the whole population, Nobel Prize winners with the rest of us. Statistically the small print includes the sample sizes in both cases.
This is fine when $n \\lambda$ is large, for then the Poisson is adequately approximated by a Normal distribution.  For smallish values or higher confidence, better intervals are available.  See http://www.math.mcmaster.ca/peter/s743/poissonalpha.html for two of them along with an analysis of their actual coverage.  (Here, the "exact" interval is (45.7575, 48.6392), the "Pearson" interval is (45.7683, 48.639), and the Normal approximation gives (45.7467, 48.617): it's a little too low, but close enough, because $n \\lambda = 4152$.)
As I understand then I will get the sampling distribution of correlation coefficient centered around the value (e.g. 0.2), which I observe in the original sample. And to test H_0 I need to take p-value, which is a proportion of bootstrapped correlations less than 0? But then it doesn't look like a p-value
Can you give more details regarding the cost function and how you're minimizing it? Also, how do you evaluate the models? How are parameter values determined from experiment? What data are you collecting from the experiments? Please revise question to include more information.
And you think of all of them as potentially response variables simultaneously (as opposed to a particular one being the response & the other two being predictors)?
**This answer is incorrect.** Consider the count for "2": because this can result only when all the 2's are encountered before any of the 1's, its probability is one in every $\\binom{8}{4}=70$ and therefore its expectation in your simulation is $10^5/\\binom{8}{4}\\approx 1428.6$ with a standard error of $37.5$. Your output of $1660$ is over six standard errors too high, making it almost surely erroneous. An accurate value for the mean (based on a different simulation with $10^6$ iterations) is $5.833\\pm 0.004$.
Do your algorithms involve some kind of randomness, or why else do you run them 10 times each on every dataset?
This seems a curious request. If you know that your questionnaire already has good credentials, you can explain that fact directly with as much as detail as needed. You don't need another layer of references; in fact wanting authorities for use of authorities puts you at risk of an infinite regress of citations.
Than probably you can fix one variable value (e.g. X_1 = 1) and plot histograms of other variables. You will see the frequencies. I do not know the out-of-box solution for such analysis. Hope this helps.
Are you doing capture -recapture?  What is the relationship between widget Xs coubts and widget Ys counts?  Without more information there seems to be no way to answer this question.
I didn't really see what you are referring to. Can you point my to the right direction? Thanks.
I hope you get many responses to this excellent question.  We need some guidance in this area.  Another approach, made easy by the R `rms` package's `bootcov` function is to save the bootstrap replicate regression coefficients ($\\beta$s) and to use the bootstrap nonparametric percentile confidence interval approach to get confidence intervals for any contrast (combination of $\\beta$s) of interest.
Thanks @peter, that is so, so helpful. I'm using SPSS. If you happen to know of any 'ordinal logistic regression for dummies' sites, please throw them my way :) I'm googling it to find out as much as I can, it was not a topic that was covered in my third year statistics course.
About the error message: there was a typo in my answer. The `time` and `group` should be quoted in the call to `amelia`. For the bounds, you can use `Inf` or `-Inf` in the place of the upper bound to get a one-sided bound.
Oh come on Stephane. This is Wikipedia, you can edit it and improve it!
You can certainly use a chi-square (but don't round off the expected value; leave it at 1000/6). Some directly relevant posts: [A](http://stats.stackexchange.com/questions/3194/how-can-i-test-the-fairness-of-a-d20), $\\, $ [B](http://stats.stackexchange.com/questions/68270/how-do-i-find-data-to-show-whether-a-shaved-die-is-really-loaded), $\\, $ [C](http://stats.stackexchange.com/questions/58438/chi-squared-distribution-for-dice-not-returning-expected-values), $\\, $  [D](http://stats.stackexchange.com/questions/14301/designing-a-test-for-a-psychic-who-says-he-can-influence-dice-rolls), ... (ctd)
cacba, you've got it right. I do not have a training set and it's not a supervised training problem. As of now I do have the rules implemented and I trigger different portions of my game based on the independent variables. But as stated, overall the AI is not responsive as the thriving value is inaccurate. I do not have a math background, do you know any examples that I could follow for the predator prey implementation?
It should be possible to match individuals in pairs for panel data because these two papers ([paper1](http://ftp.iza.org/dp7146.pdf), [paper2](http://ftp.iza.org/dp3110.pdf)) do it as well. Unfortunately the authors don't exactly state how they did it. The idea you are describing with Heckman et al (1998) is exactly the reason for using Diff-in-Diff after the pairwise matching.
If your observations are symmetrically distributed about a center, then, the determinant of the covariance matrix of your observations is the measure you want.
The CLT applies here if you want to evaluate the CDF with an approximation, which is very handy versus working out the actual CDF for the reciprocal of the average of uniform random variables. I experimented using n = 4 Uniform[0,1] variates. The maximum error as compared to the actual cdf was 0.0074, which is an acceptable error for much of applied work.
I've taken the answer a little further since your comment above.
What do you want to forecast? What textbook treatments have you looked at so far? What level of a treatment are you looking for? If you want theoretical, Brockwell & Davis is a good text. At a slightly lower level are the texts by Hamilton and by Box, Jenkens, and Reinsel. For a little more applied focus, see R. Tsay's financial time-series text. All or none of those may be what you're looking for, so it would help to expand your question to include more detail.
If you agree that it's not a good idea, I don't understand the point of your developing this algorithm.
(+1) I must admit I prefer this approach :-)
"we covered at least 80% of our population of interest" I highly doubt this means what you think it means. Do you actually mean to say, of the people approached 80% were eligible or participated in the study or completed the survey?
I understand that we're always comparing levels within categorical independent variables, but the way I was told to understand this output is that, for example, because the baseline is set as single control, for Animal_Number, the baseline is being compared to paired control- is that correct? Therefore, if I were to write a statement like 'there was no impact of animal number on survival (p<0.05)', this wouldn't be correct, as the first line of the output is actually representing the comparison of single control vs. paired control? Or am I interpreting this incorrectly?
Suggested closing this, read it a few times, and wouldn't even know where to begin turning this into something answerable.  OP is long gone.  If anyone understands this question do chime in.
Great, thank you! But your answer leads me to a second question. I am fitting a model in which each unit is a number of events per month. So the exposure is the same for all units. So far I assumed that I don't have to define the exposure option in STATA if the exposure is the same for all units. Is that right or am I making a mistake here?
Thanks for your answer. I do want to use a nonparametric model and not make any model assumptions other than its smoothness. So if I understand correctly, you are suggesting just fitting with something like LOESS and minimizing the fitted values?
What does the variable `classes` do in the code? It seems to be useless.
@G. Jay: Could you elaborate a little on what "standard and important result" this is and how it applies in this situation?
Angrist, Joshua D. and Jorn-Steffen Pischke. 2009. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press: Princeton, NJ.
Err no, the law of total probability does not tell you to add conditional probabilities the way you are showing.
@CloseToC A pretty thorough and general discussion of these relationships between correlation and regression appears at http://www.math.uah.edu/stat/sample/Covariance2.html.
thanks I will look into that
@Cardinal That's right, because I tucked the scale factor into the exponential as $\\exp(b)$, which must be positive, rather than as a separate multiplier (which could be negative) of $\\exp(kt)$.
To confirm: it's definitely not a normal distribution.
Could somebody elaborate why the answer was down voted?
Thanks as well.  Charlie's comment was useful.  Cheers.
Comment continued. The first think I would do would be to study the correlation structure of the repeated measures and the cross correlations between response and covariate series, to see if there is any structure worth modeling there. If you can legitimately average across repetitions, then you have a simple multiple regression scenario, with more or less normal variables, thanks to your averages and the central limit theory.
I don't actually. I just thought I might get different interpretation of measuring correlation. I'd love to hear feedbacks if possible about alternative correlation measures if possible
For convenience: use CTRL + R to run selected syntax.
My fault. I meant that it does not work for non-square tables with different expected frequencies. I have edited the original question.
What form does their response take? An estimate of a probability ("I'd be 60% likely to vote for him")? Some ordered categorical scale, like a Likert-scale item (e.g. "Not at all", "Probably not", ..., "Definitely")? A binary variable ("Would"/"Would not")?
What do you mean by "the probability to add a question in a forum"? Do you mean you consider things are static ? do you discretise time ? how do you discretize time ? see my answer here: http://stats.stackexchange.com/questions/11414/do-we-have-a-problem-of-pity-upvotes/11445#11445
Why  is age not in the model?
It is best to get more familiar with the workings of R before proceeding, e.g., how to easily download and install packages.  http://RStudio.org makes this especially easy.
Hi Jack, no, it will create problem while decoding. For example, if the encoded bit stream is 10 , then it can either be decoded as "BA" or "C". The encoding scheme has to be uniquely decodable.
Did you mean the normal is a good approximation? The binomial yields exact probabilities, yes?
(+1) Please add a link/full citation to Firpo et al (2009).
I believe this is a duplicate of [Given the power of computers these days, is there ever a reason to do a chi-squared test rather than Fisher's exact test?](http://stats.stackexchange.com/q/14226/930)
I imagine we'd care more about your concerns with the measurement model(s) if you specified them.  In Hendrik's case, he may have theoretical reasons for considering the two collinear latent factors meaningfully distinct.  Regardless, if the rule of thumb you're describing is the more general one for observed variables, this answer may miss his point in asking about latent variables specifically.  If you have reason to think rules for observed variables can be applied equally well to latent variables, those reasons would be useful for evaluating the applicability of other rules of thumb too.
@Bootvis: If you happen to look at the extensive comments on an answer to a recent question asking about consistency vs. unbiasedness, you will see that a consistent estimator can have arbitrarily wild behavior of both the variance and bias (even, simultaneously!). That should remove all doubt regarding your comment.
perfect! not homework ;-) I just came up with the ball question to remove the distractions from the problem I'm actually working on
In PCA of centered $n$ cases X $p$ features data the total number of PCs is $min(n-1,p)$. Imagine you have n=2 data points in p=3 dimensional space. How many reduced dimensionality is needed to explain all the variance?...
Thank you for your explanation of ACF, I am looking more for the justification to do it on the wavelet coefficients, and the signification of it...
Sorry, I find this question confusing. If you understand the meaning of a factor itself, then the individual scores are "how low or high the respondent scores on this factor". I don't see how you can understand the one without the other. Could you please clarify? The way the regression method works is explained in [this old post](http://spssx-discussion.1045642.n5.nabble.com/Calculate-factor-scores-manually-for-additional-time-period-tt1088658.html#a1088660) of mine.
...cause I would like to work out the full example.
BTW, i will use this table SET to create a decision tree model (e.g. C4.5). I think if i use a statistical model such as GMM (markov), i should take into account the frequency of their occurence.
This is a very useful way to think about it, and DFBETA looks exactly like what I am using. Thanks for the quick reply and great explanation.
On the same book is was earlier mentioned : `Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.` Maybe this is more helpful.
I believe someone else mentioned using the Lasso regression penalty.  This would be effective in your case is it reduces the number of parameters that the model uses and attempts to select meaningful covariates.
If you were to modify the question to something like "What is a way or ways to what I want, and is there a python implementation?" the first part should be sufficiently on topic. But then your question would require clarification (you end by asking about exploring, not testing -- those are very different exercises)
The answer is here: [Determining variance from sum of two random correlated variables](http://math.stackexchange.com/q/115518/29951).
Although you refer to "two null distributions," your description encompasses only one: the one created by "some sort of resampling."  What is the second null distribution to which you refer?  (The very concept of multiple "null distributions" seems inconsistent with the definition of a null distribution, so your help in clarifying what you are asking would be appreciated.)
Can you post your data?
It is 5/81+ 2*4/81.
The assertion that a particular probability is low out of context isn't convincing. The probability that I am exactly as tall as my height (whatever that may be) is zero. And, yes, it's problematic to even define height with infinite precision, yada, yada, yada...  My point is that the maelstrom of existence churns with events of infinitesimal probability happening all the time!  10,000 out of 20,000 - out of context - doesn't surprise me at all. Regardless of what its numerical probability may be.
Perfect, thanks, that's exactly what I was looking for (PITA to find that by googling for it). I'm in Excel 2010, but it works the same way in there.
Arguably off-topic, given that its focus is entirely on programming.
Thanks for those tips, IrishStat. I have picked up a few more references and looked at a lot of "extreme weather event" analyses. The latter have mostly been lacking in explicit methods. I'll try using rate, as it should be more intuitive when plotted. I now have to explain that an increase in the smoothed interval means a decrease in the rate.
but as you said in your second point, in my case, one class classification can be also applied to my case!? Since true will be patient with disease D, false will be not D ... The fact is that I don't have good false samples (not trustable), thus I only want to focus on true samples only.
+1 Beat me by 10 seconds with that one...
I buy all my books since I am a collector. If I see that it is what I have been looking for, I will pay the 30-40 dollars with great pleasure.
Many thanks I will use a loop that updates `newfit` and `forecast(newfit, h=24)`then
@Memming Thank you for your comment. Could you please explain more details? I know for nonlinear classification, data need to be projected into high dimensional space; while in nonlinear regression, why the data still need to be projected into higher dimensional space?
@MarcClaesen -- you're welcome. It's not so much a question of whether it helps, but rather a case of "how much is the HR changing over time?" You could consider the original HR from your model in the question as a kind of pooled hazard ratio over all times: adding in the covariate*time interaction will give you some idea as to whether a single HR applied at all times gives a similar story to the time-dependent specification.