tttjjj/elt_bench huggingface dataset:
@Scott Kaiser: I do not think so.  `fitdist()` is a function in the fitdistrplus package, and this is what I was using. Meanwhile `fitdistr()` is a function in the MASS package, and would not work here in this form.
Actually, I got more than 6 types and more than 5 sites. The problem is I can't remove or pool the small values. So, I guess is that appropriate to use Fisher test with monte carlo as an alternative to chi square test of independece ?
The "most natural" null hypothesis is that the compared (or implicitly compared, as in regression) samples come from the _same_ population. Hence under this H0 all their distributional parameters are assumed to be equal.
This issue is addressed in **many** posts here. *Neither* the distribution of the independent variables (x's), nor the unconditional distribution of the the dependent variable (Y) is assumed normal for the usual normal-theory regression inference. It is only the *conditional* distribution of $Y$ that is assumed to be normal (equivalently the error distribution). If the model is correct, the marginal distribution of $Y$ (that is, considered by itself) depends on the pattern of the x's and might be almost anything. The only way to assess the assumption at all is *after* the model is fitted (ctd)
I think that sounds right.  Thank you!
The $p$-value of $|z|$ statistics that are $\\ge 3$ are all *small*. They never get big again... just smaller and smaller. So if your table of $p$-values for $z$ statistics stops at three, it probably says the $p$-value is 0.000 or something like that (really meaning < 0.001), and you can simply use that to conduct your tests.
If you have a test set you might just use that for your evaluation. Just to obtain reproducible results set 10 different random seeds and average across the results. At least every time you run your algorithm you should obtain the same results.
Do you want to test the ranges as single numbers (i.e 75%tile - 25%tile) or as ordered pairs?
Even if the function used random numbers and the output could vary slightly you could always use set.seed to make sure the analysis was the same every time you ran it.
As a side note: If you really have inverted questions, I would not check if the people cliked agree - but if they checked "do not agree". If they clicked only without reading, they would never see that the question has a different meaning...
Awesome answer, just what I was looking for.  Thank you so much for the careful thought.  As a brief follow-up, I made estimates for $\\hat{B}$ and $\\hat{\\Sigma_{U}}$, and then computed the 'theoretical' test statistic for a few hypotheses.  I compared these to a monte carlo simulations, and found the mean of the monte carlo results to pretty much equal the 'theoretical' results.  So, awesome! thank you again
@AmeliaBR Sounds interesting but I am having problems imaging what exactly you mean. Do you have an example plot you could post?
Also as a note I came across this info via a blog by [AnnMaria De Mars](http://www.thejuliagroup.com/blog/?p=1390) who has a few recent posts on the US and mathematics education which may be of interest as well.
Hi Andre. Yes I am open to suggestions thats why I asked the question here. About variable names, assume exports in on X axis and GDP is on Y axis then arrows will suggest how position of each country changed over time in the graph. countries on top right corner will show high level of exports and high level of GDP and vice versa. I am interested in figuring out how countries change their long term growth strategies over time w.r.t exports share in GDP.
Thanks Henry for the answer. If I understand correctly, in the former case, you **do not** want to recognize and report the result, whereas in the latter case you **do**. Is that correct?
What do you mean "test" here? Cases that did not participate in the PCA and for which you need to compute PC scores?
Why group individuals according to 5-year increments when you could simply have years of experience as a continuous variable?
If your dependent variable is binary, binary logistic regression is the way to go.  What do you mean by "1 independent variables (Factors)?" ... What variable presents a problem with setting the reference category? ... What specifically "doesn't make much sense"?  Also, how about showing your commands and your output so volunteers here can help you interpret and troubleshoot.
Presumably you mean not *Lebesgue* measurable.
Of possible interest: [Why are Jeffreys priors considered noninformative?](http://stats.stackexchange.com/q/7519/10525).
+1. Regarding the second question, how about a "goodness of fit test" (Classical or Bayesian)? This is the closest statistical concept to the OP's question I found in the sense that even if we select a model, a goodness of fit test might detect a poor fit.
In way 2, isn't my sigma.square higher in the second way of computing the mean ? My question now is actually is the expected sd(rnorm(n,mean.1,sd.1) + rnorm(n,mean.2,sd.1)) == sd.1 ?
Big data is probably one area where bias variance decomposition is not helpful - data quality and data management are more important.  This is because we cannot hope to know every data point or even special cases - just too many of them
I agree quite strongly with the main points here, but the first sentence is oversimplified. First, outliers can be outliers, but still consistent with even OLS regression, possibly after transformation.  Second, it's the same story with heteroscedasticity, where there are various remedies, including also using weighted least-squares. As I remarked earlier in this thread, it is not helpful to conflate models and methods of estimation.
Just find out how to get pooled (chunk; composite) tests for race from PROC GENMOD or PROC LOGISTIC in the context of multiple imputation so that the right covariance matrix is used.  PROC MI explains how to do the latter.
In other words, $\\leq$ 100 PCs cover the whole variance in a data set of size (100 x 1400). Mathematically, there cannot be any further variance in this data set. Things may differ, if you look at the variance in the ground population that is sampled by your data set.
It is a pity that there was no slider or a numeric box in the form. It is always better to truncate your data in the analysis than in the data collection. A classic example is high blood pressure where studies running for several years risk that the criteria for high change during that study period.
Thanks, that are answer what i looking for.
First off, apologies for not noticing the silly mistake. I am not a huge fan of fitting these exceedingly complicated functions. Think of what happens to the variance in your answer if you are off by a bit. I would still prefer a simple model where you set $X_{4} = X_{2}^{\\beta}$ and then estimate $\\beta$ in a subsequent step.
You need to know (or assume) more about what's going on. In some cases, the units within a packet might tend to have very similar weights; in other cases, they might often be wildly different.  Would it be possible to destroy a small representative sample of the packets in order to assess this?
(+1)  This is a great idea generally.  For large correlation matrices, however, there are so many statistics and so many of them can simultaneously be spuriously large that it usually pays to adjust.  Otherwise you wind up chasing a large number of misleadingly "significant" correlations that just disappear in the hold-out data.  (Run a simulation with, say, a few hundred draws from 50 uncorrelated standard normal variates.  It's an eye-opener.)
@Theis: Yes, but my question is whether you can do SIS to also get points distributed according to $\\pi$.  In my above procedure I would expect that after some 'burn in' time the particles would be drawn from the stationary distribution equal to $\\pi$.  The difference then, between Metropolis, is that the expectation is given by a weighted distribution rather than an accept/reject distribution (likely with many duplicates).  This difference would be in some ways similar to vanilla importance sampling vs. vanilla rejection sampling.
I'm not sure I understand what the problem is... why do you want to take the ln of the values exactly? Because the distribution is positively skewed I suspect?

Also, to me using 6 dummy variables in this case seems to be something of an overkill. You have the privilege of having your scale start at 0, be ordered, and have equal intervals; with 7 different categories, I just might forget about dummy variables and use it as a continuous variable...
I wrote an example of how you can do it: http://rpubs.com/emdiazsal/6194
+1, nice overview. NB, a slight shortcut might be to use [?qqPlot](http://rss.acs.unt.edu/Rdoc/library/car/html/qq.plot.html) w/ `distribution=weibull` from the car package, which will fit the paramaters via MLE & make the qq-plot in 1 step.
`My sample is unbalanced, i.e. some cells do not have data at all` So highly unbalanced design is known as empty cells design. I think you ought to read more about it. The problem here is not coding.
Yes, filler text.
please provide both example data and your desired outcome.  otherwise explain why [this post](http://joelcadwell.blogspot.com/2013/01/if-spss-can-factor-analyze-maxdiff.html) doesn't answer your question :)
@Karl In principle, some dependence among failures can be accommodated.  One way is through [fault tree analysis](http://en.wikipedia.org/wiki/Fault_tree_analysis) and event tree analysis (*ibid.*).  One has to have a complete, accurate model of the entire system for this to succeed, and even then unexpected rare exogenous events will not be accounted for.  The moral is that any honest failure/error rate estimate will be heavily qualified with assumptions.
"*I have read the linear regression requires that the input features be independent.*" --

It's not true; however, they mustn't be too strongly dependent (perfectly correlated or close to it is a problem). If the two are not too highly correlated it *might* make sense to include both.
Since neither p-values or AIC were designed for model selection, they dont have assumptions for model selection.  Both were designed to do a single comparison, think about how many comparisons take place in a stepwise regression, do you really think that the "best" step is taken each time?
Thank you @javlacalle
The formula you gave is not an approximation.  It is the exact formula for the variance of the difference between two random variables.
First one is NEW, second one is ALL as I understand. So, first is a derivative of second by time.
I agree "-1 df" doesn't seem sensible output for any statistical model. Think you may need to email the authors of the package.
Thank you for your contribution.  A short summary of your approach would be appropriate (and welcome), especially because the link does not go to a generally recognized, authoritative source about statistics on the Web.
Andy, I've edited my post to address your comments.  Also, I don't mean to suggest that causal inference cannot be done outside of the context of a designed experiment.  Nonetheless, it is more difficult and less certain in observational studies and we shouldn't look to model building procedures to help us with that problem.  Rather, we should try to better understand the problems for which we are attempting to understand causal relationships.
1) How can $\\text{abs}(|t'|-t_{crit})$ be negative?
2) If I understand correctly, it means that the procedure only makes sense in a hypothesis test setting, where I have some null hypothesis about $\\mu$?
Hi @Avraham. I don't want any value outside the max and min in the synthetic dataseries generated. I would like to get n random lognormal synthetic dataseries from the original, but i don't want any value outside min and max of the original.
see [here](http://stats.stackexchange.com/questions/45588/variance-of-a-bounded-random-variable#comment88534_45588) and [here](http://math.stackexchange.com/questions/83046/maximum-of-the-variance-function-for-given-set-of-bounded-numbers)
@Gavin: I agree; in this case with orthogonal predictors, examining each individually will give the same coefficient estimates, but inflate the error variance (assuming there are effects). I'm not sure how that's supposed to deal with overfitting in any case. Model reduction *might* be appropriate to improve predictive accuracy, *if* that's what you want. The next step if this were an industrial experiment might just be to bump up the temperature & splash in more acid, leaving time alone; then monitor the process to confirm it's doing what's expected.
Your random reordering reminded me of this AI koan: In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe." "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play." Minsky shut his eyes. "Why do you close your eyes?", Sussman asked his teacher. "So the room will be empty." At that moment, Sussman was enlightened.
My first thought about the publication bias was that the OP is concerned about the data accessible via literature study, not whether about how to publish the results of the meta-analysis.
Thanks a lot for the input, I will definitely give it a try with zoo() or something else later, FYI , here is a link to Professor Hyndman's blog [link](http://robjhyndman.com/hyndsight/forecasting-weekly-data/)
@StasK I wasn't expecting the community to write papers for me, I asked for pointers in the right direction. Thank you for letting me know that this is a substantial undertaking, though.
Nice note on how having a constant term in the regression matters.
I'll take a look, and also try to find a connection with the martingales results. The bound for $U=(1/n)\\sum_{i=1}^nX_i Y _i$ is so easy ($\\mathrm{Pr}(U\\geq \\theta/2+\\epsilon)\\leq \\exp(-2n\\epsilon^2)$) that it's tempting to connect this with $Z$ using some kind of conditioning.
Data frame with 98 observations and 107 variables sounds like simple table (no correlation matrix). Please clarify!
It seems to me the novelty of your work will lie in how you define "prototypical" (of which I am not aware of anything directly related). It reminds me a bit of [Gelman and Pardoe (2007)](http://www.stat.columbia.edu/~gelman/research/published/ape17.pdf) in which they use Mahalanobis distances to define "average" effect sizes. Those same distances can be used to construct potential "protypical" individuals.
After, although the more thinking you've done before the experiment the more likely you are to gather the right data.  While the analysis is after the fact, you obviously can't do it if you find you've neglected to collect some crucial bit of info.
@MichaelChernick to make thgings simple I fix the threshold as a constant positive value, above the initial CUSUM sequence (so the CUSUM will increase in response to a small change  in the mean of a process, and eventually cross the threshold). The drif is also fixed and in this case it has a negative value to be able to model the increasing CUSUM.
I have been stuck in this p value = `NA` problem for almost the whole day. I just wanted to know what went wrong. It would be helpful, if you can give me some help in this problem, because I think I am still not grasping the concept of chisq test well enough.
It took me a while to find that one (I knew I had seen it somewhere - it is missing a crucial tag though!) I'm thinking that will solve your problem, but if not just clarify how this is a different situation.
Randomly splitting a dataset into two parts is not only valid, it is recommended. You want the two samples to be from the same population.
Hm..I think @NeilG is right because E and F are also independent (they're the parents of D). Therefore, if I'm not mistaken P(E=i∣F) will become P(E=i)
Beta distribution is good for ranged data.  Use it instead of normal.
@JohnSmith: You are alluding to the point I was driving at. There is no indication of how to measure "optimality". What you are effectively doing is introducing a different metric (distance function) to measure "quality" of prediction or fit. We need more details from the OP in order to get very far, I suspect.
one can also use this to generate discussion on spurious correlation
For your first question, it's not clear what is to be compared.  Are you considering each mutant one at a time?  In the phrase "difference between means": "means" of what?  "difference" between which and which?
It could remain constant over some range of $\\lambda$, but it is true that it's nondecreasing as you increase $\\lambda$.  To amplify this comment, consider the case where $\\omega \\in \\{0,1\\}$.
@IrishState residuals vs observed will show correlation. They're more difficult to interpret because of this. Residuals vs fitted shows the best approximation we have to how the errors relate to the population mean, and is somewhat useful for examining the more usual consideration in regression of whether variance is related to mean.
"Is this sample size sufficient?" ... sufficient to do what, exactly? You can compute correlations with fewer data points.  And why would you reverse-code anything?
@pedrofigueria I have seen this wiki before. I know the equation "y = a + b1x1 + b2x2 + b3x3 + b4x4 + .....". In wiki, in the place where it's written, the least square estimates, `^b`, can now be obtained: why has the first column of `X` matrix been taken `1`? I can understand the other values corresponds to `x` and `y`.
Statistics works by training. The idea is that you need to first identify (by yourself) a set of points satisfying what you are looking for, then try to train a statistical procedure to do this job for you.
Thanks for the answer, I was losing hope and even learned a bit of BUGS in the meanwhile :) I don't have repeated measures, just individuals within countries. My x1 is time-related, so it is continuous. Does your formula change then? I had my multilevel course in MLwiN and I am a bit confused about how MLwiN formulas translate into MCMCglmm arguments. For instance, what does rcov correspond to, and how is this different from when you specify individual variance in the random part. That might be too broad a question, just a working example I can depart from is great already.
I would like to add that one thing that may seem confusing, is the use of covariates to "control for" effects in regression analysis. Covariates doesn't control for anything, but they try to ADJUST for them.
@Tom Reilly, thanks for your help. Could you explain the result a little bit? Coz I really don't understand the equation or how the equation was achieved. And I didn't find a detailed tutorial of autobox either.
just great, thank you !
The fact that the system is "underdetermined" does not imply it has any solutions at all. The question is about existence of solutions.
The 'forecast' package has a function called 'accuracy' that does this kind of test automatically. I'll use 0.7 of the data to train the model, so i can get some indicators. I'll also look at ETS models. Thanks for you answer!
@user1322296 I think you are looking for confidence intervals for predictions. Have a look at this answer: http://stackoverflow.com/a/14424417/1412059
In PCA you can look at the composition of components, so you can see what attributes were mainly used for creating first PCs. Nevertheless it is useless for data with complex or subtle dependencies.
I don't think a chi-squared analysis is appropriate here. 1st, your data aren't really separate, so you don't want to treat them that way. Prima fascia, you want to use a logistic regression model that can take all variables into consideration simultaneously. However, the key issue in your situation is that for every favorable verdict for the prosecution, the verdict for the defense is necessarily unfavorable. IE you don't have independence here, which makes your situation more difficult / interesting.
Could you please be more specific about the "files" and what you mean by "whether files are different". E.g., in each file do you have a univariate time series and you'd like to know whether the trend of each of these time series is different or something similar? Or how do you define different based on a file? It would be best if you included an example.
i think only one of the classes in the Iris data set is linearly separable. (From the OP's Question, i think he's after data sets w/ *only* linearly separable classes).
I found it interesting but I don't think it is at the level expected by the OP at all.
Don't ever mention PLS in my presence if you want me to respond again :). What you have then is a MIMIC (multiple indicators, multiple causes) model, and that's what you should look up. You should be able to set up your this SEM model in any decent SEM software. To identify your latent variable, you need to have at least two "reflective" indicators of it (e.g., measured GFR and creatinine level). While psychology uses "formative" and "reflective" indicators terminology which I find awkward, sociologists use "causal" and "effect" indicators terminology, so you might want to look these up, too.
Also, I'd give you an upvote for this, but I clearly don't have enough reputation yet.  *upvote here*
Although, I'm not sure how R is any less state-of-the-art than SAS (and with regards to many procedures, I am under the impression that R is _more_ state-of-the-art than SAS). My guess is that a lot of SAS users don't know about that though...
Multiple linear regression.
These are good points, and, in my opinion, are very relevant in biology.
thanks for responding.  I'm not quite sure how I would implement your suggestion because, given comparisons AB and AC, the BC comparisons are redundant information (i.e., they are completely defined by the other information in the network).  So, if I were to do a (multivariate) contrast-based meta-analysis where my outcomes were the AB and AC studies, I would not be able to add another variable that represented the BC studies because the estimation procedures would crash.  Unless I'm misunderstanding your suggestion, of course . . .
I don't know, but you have some very large coefficients there with enormous standard errors. [Separation](http://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression/68917)?
I never got back to answer my question, but the problem apparently had to do with two parameters that I had. The thing is that in my model, the ratio between them was the observable (say, $\\theta_1/\\theta_2$) but I foolishly set both parameters separately (say, $\\theta_1$ and $\\theta_2$). This generated a lot of dispersion in $\\theta_1$ and $\\theta_2$ because there are an infinite number of possibilities if the ratio is the ('fixed') observable, and this caused the effective number of parameters to explode :-)! Thanks for helping me in checking that everything that I was doing was ok!
One reason is I want to plot points on the survival curve outside outside the time of my data.
+1. To me, the strange part is that the fitting and the simulated predictions are all done within the body of a single function. Normally, operations like this would be done by first computing the fit (returning `beta` and `M`) and then creating *numerous* iid simulations based on this fit. (Putting them in the same function would unnecessarily cause the fitting to be repeated each time, greatly slowing down the calculations.)  From these simulations, one could recover (*inter alia*) prediction intervals for nonlinear or very complicated combinations of the responses.
@CarrKnight: this is a good example of what I meant by "tuning the model". Otherwise, with static and known probabilities, the answer would be trivial. Another possibility, less sophisticated than your suggestion, would be to assume static but unknown probabilities, which have to be estimated and optimally exploited for a given finite path. Further, one could allow for arbitrary step-lengths instead of one-gridpoint moves.
Joaqin, welcome to the website. You might want to raise this question on SEMNET, a mailing list on basic SEM education.