tttjjj/elt_bench huggingface dataset:
If your data are only ordinal how do you know how skew it is? Are you implicitly assuming nearly equal intervals? (Edit ... I see from your comment that perhaps you do have additional information that might make it correspond to apparent skewness) When you say "whereas the rest is quite normally distributed from 1-8" - you can't mean normally distributed. You probably mean something like 'sort of bell shaped, reasonably symmetric'
Unless your data are artificially generated by a particular, known distribution, I can guarantee that any reasonably effective distribution test will reject them!  Nevertheless, it can be meaningful to use a goodness of fit statistic, like the KS statistic, to measure the deviation between the distribution of your data and any ideal distribution (there will be no problem with ties or discreteness, either), but the p-values are worthless.
To me at least it is not entirely clear as to what data you have and what you are trying to achieve (e.g., which data exactly would you calculate the CV for). Perhaps you could clarify your question?
(+1) Does this package also implement regularised regression (shrinkage) for negative binomial family? Any idea? Any other packages you're aware of?
In the event $u$ is a _deterministic_ function of $(t,M,Z)$, you are simply looking at the _projection_ of the posterior distribution $p(t,M,Z|x_0)$ using the function $u$. (I would not speak of _predictive_ then...)
Could you add a few words about *why* this is true and perhaps what $n$ must be to be considered "large"?
Accepted! Thanks!
That's a very good description what the scipy code actually does. The part where it raises the error is when it takes the inverse of the sample covariance matrix. In the univariate case without variation the covariance matrix has just a single zero element. statsmodels has in this case mainly the plotting code, the kernel density is calculated by scipy.stats.gaussian_kde.
@mpiktas, I never did see Hypercube, and it does not sound like I should :-) I saw The Cube some ten years ago and found it pretty entertaining.
That sounds good. Thus my guess was right.
Hi Peter, The data was negatively skewed and peaked. Squaring the variables did the job nicely. I've done the paired T tests and that tells me something about the variances between customer pereceptions of the two companies. What I'm intereested in is understanding how the competitive position effects overall performance rating. I'm making an assumption that ratings of performance along a range of variables is relative to the competition each customer experiences. I was planning to use the differences between the customer and competitor scales to do a regression analysis.
I am a bit murky on this at the moment, but I think that in this case "living in a simulation" is not independent of "creatures have eyes", and you have to take that into account when finding the maximum entropy distribution. Take a look at http://en.wikipedia.org/wiki/Principle_of_indifference for example.
And no, I don't have a citation.  If you have a large effect it won't matter if there is a little bias in this measure.  If you don't have a large effect make sure you don't make big bones out of the "significance" of any test.
Can you clarify what you mean by 'augmenting' the training and testing set. You should never do any CV on the test set, it is used solely to get an unbiased estimate of the model.
@Yuriy, Don't know what makes it incorrect if the first plot is not a failure, but if necessary, you can add `DF <- rbind.fill(data.frame(outcome=1),DF)` before computing `DF$x` and `DF$y`.  But then I question the meaning of what you are really plotting.
+1 for the suggestion of using simulations. A couple of blog posts about simulation in SAS that may (or may be not, depending on your SAS knowledge) be useful: http://blogs.sas.com/content/iml/2012/07/25/using-macro-loops-for-simulation/ http://blogs.sas.com/content/iml/2012/07/18/simulation-in-sas-the-slow-way-or-the-by-way/ http://blogs.sas.com/content/iml/2012/06/06/tips-to-make-your-simulation-run-faster/
Thank you for the guidance; I've got a better handle on what I'm capable of, now that I've poked around in R and not made too much of a mess.
If you have a vector, `dim` won't work.
Thank you very much for the answer and for your great package :) In situations like this, is there any parameter that I can tune to ease out the effect on the seasonally adjusted series? If I throw away the last 7 points (of 24 points of the last day), then the forecast comes back to normal. It does not make sense (in practice) when I have 50 days of regular data and the forecast for the whole next 5 days drop just because of the last few points. Or do you suggest any better model/approach for such kinds of data? Thanks.
The non-central t distribution is not some shifted version of t: see the [Wikipedia article](http://en.wikipedia.org/wiki/Noncentral_t-distribution) for details.
The first paragraph explained very well what was unclear to me, I couldn't find that piece of information (about the scaling).
tradition and Csiszar say $L_2$, Aggarwal and a few others $L_1$ or $L_.5$ or ... What to do ? Without solid reasons, I guess it depends on your mindset / your prior beliefs.
300000x300000 matrix? Wow. Can your computer keep it in RAM memory? Honestly, I doubt that anyboby in the world has ever done MDS on such data. You'll probably be the pioneer.
Do you have a specific method in mind for measuring similarity between two sets of gene targets, or is that part of the question?
Correct, to understand the behavior at a given sample size, you take the same sample size, sampling with replacement. Exactly what is resampled varies from situation to situation and even model to model.
The discretised hazard is zero, except at event times. This indeed gives the largest contribution to the likelihood if a
discrete hazard function is supposed. You might want to interpolate between any two estimates assuming, for example, that the hazard stays constant.
The error is probably the dependence issue. If the time trend was the appropriate model, then all the time dummies would line up on a straight line when plotted. To the extent that is not the case, then there is not a constant time trend. Typically, we use the more flexible model with time dummies. The time trend adds no information that you can't get from the time dummies alone.
I can define a uniform distribution with mean 0.5 and standard deviation 1/12. It is perfectly well defined, but it is not the most natural parameterization. There is nothing about standard deviations that implies normality.
+1 Tukey's original boxplots were oriented towards pen-and-paper calculations and thus were based on the "hinges" (or "fourths") rather than quartiles.  The hinges are medians of the two sets of data created by splitting the original batch into two equal-sized pieces at the median (including the median in both pieces when there is an odd number of values). This ties them more reliably to actual data values than many of the interpolated quantiles in `R`.
You must first identify an ARIMA or AR(p) model for the x variable and get white noise residuals. Then you filter y series by this same model and get "residuals" which probably are not white noise. Then you do bivariate cross-correlation analysis with "residuals" of x and y series. Model for x series can be identified by ACF/PACF. Here is link to this method: https://onlinecourses.science.psu.edu/stat510/node/75
Note that $\\Pr(X\\le x|X\\gt Y)$ = $\\Pr(\\max(X,Y)\\le x)$ = $\\Pr(X\\le x\\text{ and }Y\\le x)$ = $\\Phi(x)^2$. This agrees with your result after you perform the final integration using the substitution $z=\\Phi(y)$ (no Monte-Carlo needed).
Were those respondents chosen randomly from the pool or were surveys sent to the entire pool and the respondents are those who chose to return a survey?
Although this is correct and helpful in general, frankly, I do not see how this answers any of the OP's questions. Aside, it seems to me that the call to "rev" should be removed.
And I was thinking of it more from the software end.  A user builds a model that gets 90% accuracy on training and 75% accuracy on testing.  They then tweak some knobs and settings in the software and get 80% accuracy on "testing."  They go back again and do more tweaks and get 85% accuracy on "testing."  But this so-called "testing" dataset is no longer out-of-sample, and the model has been overfit to it.
If this is for self-study or homework, your question should have the `self-study` tag. See http://stats.stackexchange.com/tags/self-study/info
Hi drknexus, thank you for your reply. Makes sense. This is what I have done as well. I'll wait a while before marking it as an answer.
The outcome of SVM training is not subject to change for a given problem. If the training problems for two solvers, e.g. svmlight and libsvm, are exactly the same, their respective solutions should be too. The solution to an SVM training problem is a global optimum and is therefore unique. If you get different solutions, there *must* be something else happening to the data.
Interesting, done some research and found this to be a really complicated problem, and quickly becomes intractable. I think your solution is the best I've seen for practically handling this.
First make sure you need to combine different prediction systems vs. choosing a single more comprehensive method (e.g., penalized regression with many predictors in a single model; random forests).  If you really need to combine multiple risk predictors to get a better risk prediction, search Bayesian model averaging (BMA) or http://www.bepress.com/ucbbiostat/paper222.  One form of BMA weights competing models by the probability that each model is the "correct" one (if such a thing exists) using Bayes factors (see Adrian Raftery's work).
This image has been posted before on this site, so I'm pretty sure it is not original to you, Nate. Please include an attribution of its source in your answer.
+1 The first equality in this explanation is perhaps the most important, because it exposes the abuse of notation inherent in abbreviated expressions like "$X\\le t$", which in a more revealing form should be written in the way you have first expanded it.  The "$=^1=$" part doesn't make much sense--precisely how is one supposed to read "$X(\\omega)=c\\le t$" unambiguously?--and it's a trivial step anyway to move directly through $=^2=$ to the conclusion.
Sorry i think i misread --- looks like you've done this and now want to deconstruct analysis by looking at 'simple effects'.  the difference between encoding at each level of speed.  I assume your comparisons are not 'planned comparisons'.   post hoc analysis corrections would be appropriate in this case.   you would need to /9 for bonferroni correction.  Sidak is a little less strict.
Yes, you specify case 2: blah blah ; The advantage over if/elseif is improved readability.
The Weibull is a continuous distribution so that the probability of getting exactly zero has probability zero. If you're getting many zeros in your data, that's an immediate clue that the Weibull is inappropriate. At any rate, your data look like *count* data (or at least, are discrete) and so a Weibull is probably not the best choice.
I don't follow this. One point to the pilot is to obtain preliminary information such as useful estimates of population variances, which can then be used to design a full study. The pilot is intentionally made small so that no resources are wasted in collecting unnecessary information. What prevents you from conducting such a pilot study, then?
One way to understand the relationship between the log transformation of two values $x_1$ and $x_2$ and their relative error $e=\\frac{x_2-x_1}{x_1}$ is to note that $\\log(x_2)-\\log(x_1) = \\log(\\frac{x_2}{x_1}) = \\log(1 + e) = e + O(e)^2,$ showing that when $|e|\\ll 1$ (which frequently is the case unless changes are dramatic), a difference of logs basically *is* the relative error.
The opening remark of your answer gave me the idea of starting an "Annotated Fisher" wiki...  An idle thought, really, since I've never done anything remotely like it.  In particular, I have no idea of what it takes to set up and manage a wiki, and I have even less of a clue of the legal/copyright issues that would need to be dealt with to get such a project off the ground.  I agree, though: it would be a truly invaluable resource.
Could you give us more details on what you are trying to test and what your experimental design is? I'm wondering (1) why you want to include an interaction term without the corresponding main effects (i.e., `yld ~ rep + parent1:parent2` instead of `yld ~ rep + parent1*parent2`), and (2) why you consider only one contrast for the interaction since this term has 14 DF in your model (unless you're interested in a specific post-hoc contrast).
What's the sample size? If it's greater than n=80 (so that n/4>20) then you may want to inspect higher lags, in particular lag 24 and possibly lag 36. Have you calculated a t value for the autocorrelation coefficient at lag 12? If so, what is it? Lastly, what do the blue lines in the ACF represent? Since you're trying to identify a tentative model, make sure that these are based on Bartlett's errors.
@RobertF: Could you please send me the copy of Ren et al (2010)? My e-mail adress is: marco.munda@gmail.com. Thanks in advance.
So, the choice of the "right" clustering method (which is very important choice indeed) is not driven by the wish to get the _known_ number of clusters. It is driven by the nature of your data (such as numeric? binary?), assumptions about the distribution/shape of clusters (normal? globular? irregular?) and their juxtaposition (demarkated? fuzzy?).
Yes. It's not only possible, but indeed highly likely. This is an utterly standard feature of significance tests. The main function of significance tests is, arguably, to avoid being fooled by chance-like fluctuations in small samples, but that is not your risk here. It is best to focus on confidence intervals for your parameters. (EDIT: You already have this kind of comment and links to relevant questions, so why is this being asked again?)
The inverse-chi-squared distribution has just one parameter $\\nu$. Can you clarify the meaning of $r$ and $s$?
+1, good info, but it's also worth mentioning that if your estimator is unbiased, you can have a saturated model & still have an estimate of the parameters, if that's all you need. You won't have an estimate of the variability or be able to do inference. However, in some cases where there are many effects to estimate & data are sufficiently hard to get, saturated models are sometimes used. So eg, in this case, you could get an estimate of the function w/ the quadratic w/ 3 points. I don't necessarily mean that it's a good thing, but that is the real lower bound & the reason why.
Not clear on uniform distribution/CLT/normality and rank tests in this context.  But I recommend the proportional odds model for this problem.  It generalizes the Wilcoxon-Mann-Whitney-Kruskal-Wallis test.
thanks for the answer! I have a question about it: you say we assume the "sub-experiments are independent and normally distributed with the same mean". My data are not normally distributed...they're essentially binomial (clutch size of a bird that lays only 1 or 2 eggs in most locations, and very occasionally 3 eggs in some areas). It didn't occur to me to include that - apologies for that. Any chance I can still use this method?
@ilir Your replacement proposal will bias the results as well as reduce estimates of variability. It is desirable to avoid both those errors. Thomas, I would like to suggest that *calculating* the mean of the measurements is not of concern. What you want is a good *estimator* of the true value of the physical quantity. Although it might seem strange, obtaining a negative mean for the data is valid. Its meaning can be appreciated when accompanied with an estimated standard error. For instance, $-0.12\\pm 0.31$ would indicate your measurements cannot distinguish the quantity from a true zero.
Implemented a version of this in an easy situation (Normal-Normal prior with very small variance for observed single data point, and exact MC sampling of the posterior) and, in that case it works very well regardless of the true value of the parameter, so perhaps the method is better than I thought.
As for your own suggested p-value: at the very least, it's not so obvious that you can just plug in your estimate (211/230) for the population proportion in the cumulative probabilities.
I think in this context it means each bit is independent _of the previous ones_. But this is not correctly formulated.
I am not sure to figure out what your data look like. Maybe it would be benefit to make it clearer. However, if your variable of interest is of the type "time-to-event", then an analysis that takes time into account is definitely more informative than an analysis that only considers the number of cases. One reason for this is that censoring often occurs in such a study and a "time-to-event analysis" is especially designed for that setting...
Could you give us your model? I think you're a little confused about your hypotheses: Each hypothesis represents a different class of models; the third just happens to be physically impossible.  Even if you tested H2 against the composite of H0 and H1, you're not testing "the model is wrong" against "the model is right" unless H2 indexes every possible model except those in H0 and H1...
Depending on the length of your data set, you could try to look for look-ahead value of bias first (say you have 100 points take 50 for evaluation of trend by say HP on the whole sample and only on the first 50 points, then move by one the window from 2 to 51 and so on, look at the accuracy of your edges). May be in your case it is not so huge as you imagine.
How would i set it up to do the interaction term (Age x Diagnosis)?
This does not seem correct. It does not account for the fact that the data are not independent.
$\\beta$ is the true unknown model which is *assumed* to be fixed and not random (see point 1. in the answer above).
No, because I do not think you are dealing with the case that your Poisson parameter is a function of time. For all you know it is continuous, but you are simply missing observations (like a train passing in front of the window doesn't mean what is behind it stopped). Which is why I think you are better off analyzing the data you have over the timestamps you have. Dropbox is blocked here, but shave off a bit from either end of the non-gaps, to prevent distortions coming from the return to visibility, and analyze rate-per-VISIBLE time.
@Crazzzy12 I added an answer.
AFAIK Benford's law says that in many real-world variables, 1 should be the first significant digit about 30% of the time, not 100%
+1, but I have to say your answer is math heavy for me,;-). Anyway, as I read into your answer, 1)bias is used with some random processes, 2)and it's just a guideline for us to analyze the model, since we don't or can't know what the bias is exactly, because we don't know what the true parameter is. Am I right?
When $p=1$, Wishart is Gamma, and the sum is not Gamma.
The textbooks usually are clear that the scope of validity of these tests depends on the *counts* rather than the proportions. With $N=10^8$ and $p=10^{-4}$, the counts will be around $10^4$, which is quite large indeed.  (I do presume you mean $10^{-4}$ and not $10^4$.)
(by the way, with the nice and superfast help of the community (COOLserdash), I fixed the code.
Hi.  Thanks for your answer, it is helpful.  I have one more question. I will plot the CI for the difference, but I would also like to plot the individual proportions next to each other on a bar graph. People usually expect to see the 95% CI on each bar.  Is it meaningless to plot this?  If I do plot the individual CIs for each bar is there a limit for the expected to overlap (e.g. by 25%).  My CIs overlap by approx 40% but the z test still shows significance.
@Dilip Sarwate, fixed. Thanks!
Regarding your main question, you need to clarify what "the predicted probability of them ranking high" means.
Not quite sure what you're asking.  The model you wrote down is precisely the model for the random effects model; the correlation structure is induced by the random effect.
Thanks you Ariel :)
Hey Gung, i have added the ROC curve and the info regrading the aggregates,please check and guide me.
I think you forgot the figure. I also think it would help to clarify what you mean by diurnal or what your time unit is. It sounds like you have daily data, but to me diurnal means  recurring every day, which means you have higher frequency data (like hourly).
Mili, you will find several dozen closely related--and possibly helpful--threads by clicking on the [tag:change-point] tag.  Keywords for a general search of literature and software are "changepoint" (which refers to a sudden change) and "online" (which, in statistics, often refers to applying procedures to a steady stream of data). A related search term would be "quality control" and associated technical words such as "cusum" and "control chart."
Ok, thanks. I think I assumed that if ANOVA rejects H0 then you could use multcompare (a multi-comparison test) to see which of the means are different, but I suppose its possible that you can't infer anything in general about multcompare rejecting H0 if ANOVA rejects H0 or vice versa.
Thanks again.
Hi Andre, I simplified the example in order to present it in "text book" style but at this point I have no idea whether this can be solved using off the shelve methods.
Closely related: [What is the best way to identify outliers in multivariate data?](http://stats.stackexchange.com/questions/213).
First, you're analysis would be better if you used a metric more like: "# of deaths per 100,000,000 miles flown" (or something like that)... As for calculating the probability of an air accident, it depends on how accurate you want to your estimate to be. As a first-order approximation, you probably *could* use your suggested method and your result would probably be more accurate than just guessing "0" (but that's not saying it's the *best* approach available--just that it's probably better than, well, nothing).
@Dov (1) Something like what fraction of randomly shuffled versions of decision obtain better accuracy (or its parametric version). (2) If you make N LOOs of the same deterministic algorithm, the sd of results will be zero and useless; in case of stochastic one, this will be some approximation of uncertainty of mean error.
Doesn't the PR-curve require some more fancy integration? See: http://mnd.ly/oWQQw1
Thanks for your answer. I did not understand one point: can the $A$ matrix be arbitrary? I did not follow the argument on factors/dummies/contrasts.
Be careful, note that survey weights may often be doing things other than just dealing with a complex survey design.  There may also be non-response and post-stratification adjustments in the weights.  Remember, the weights are for *estimation of population totals*, not just the inverted sampling probabilities.  Also, did you read the paper in the link?
Selam Ufuk, 1) and 2) sound correct. In 3), I did not quite get what you mean by "_So, for each sample of the process G, we have a new directed graph Gn whose nodes are connected to the corresponding θi nodes and given all θ nodes, these N samples are independent from each other._"
Thank you @chase, this pretty much answers my question. I was wondering if it is possible to do this without the inbetween step of creating the "extra" data-frame.  Also I used to think that `cbind` is more efficient.  Is it not?
Out of curiosity, what does this product operation model?
Hello, the simplest way is to write an R program with two loops.