tttjjj/elt_bench huggingface dataset:
It depends what you mean by 'confidence interval of that set of data' and also depends on your model for the serial dependence(/the model for the joint distribution of observations).
I agree on disliking log(x + 1) as a transformation on various grounds, but the last suggestion here is dubious, if not dangerous. ln(zero + epsilon) may sound more conservative than ln(zero + 1), but the smaller epsilon is, the larger the negative logarithm created. Using log 10 so the numbers are easy, log10(1/1000) = -3, log10(1/1000000) = -6. log(x + epsilon) thus is all too likely to create outliers out of zeros, and outliers whose values depend crucially on an arbitrary choice of epsilon. This is why 1 was suggested in the first place, presumably.
I am trying to set something like this up, mainly for the purpose of making visualisations of data on international development and particularly economic and education policy. The beta version is here: http://devtimelines.appspot.com. You may have in mind longer term historical events (this only goes back to the 1970s) in which case this will be less useful for you. It doesn't yet allow the data to be accessed externally but I intend to add this facility (especially if there is actually a demand for it!)
I don't think it's possible. I think I will lost a lot of the information contained in my data
Great idea!  Now who wants to publish the results?
One thing you may want to do is "stabilize" the proportion to handle the log transform, something like $\\frac{b+a}{c+2a}$.  You may also want to add the logit itself as the predictor $\\log\\left(\\frac{b+a}{c-b+a}\\right)$.  This is simpler than adding 3 predictors.
Although it is unclear what you mean by "statistically differentiate," it sounds like material on [peak detection in signal processing](http://stats.stackexchange.com/questions/36309) and [change-point methods](http://stats.stackexchange.com/questions/tagged/change-point?sort=votes&pageSize=30) would potentially be relevant and useful.  Possibly some related material also appears in [threads referencing "genome"](http://stats.stackexchange.com/search?tab=votes&q=genome). Perhaps, after perusing some of these, you might be able to edit this question to clarify further what you need.
I find this data are little hard to visualise (if you can excuse the pun).  Is it stored as a rectangular dataset, and if so, how many rows and columns (and what are they)?  If not, what dimension array would it be stored in? Also what is the time series aspect?  Thanks.
you sample with replacement - that is, a subject can appear more than once in the bootstrap sample. Example in R;x<-1:10;
sample(x,10,replace=T)
Well, I have finally figured the way to deal with my data - to use standard GSEA (Gene Set Enrichment Analysis).
Thank you so much for your help!

As I'm not so familiar with the R syntax, this works only for fixed group sizes, doesn't it?

Is there also an existing function for comparing the permuted values with the observed one and giving back the p-value?
It takes only the most casual acquaintance with WinBUGS to answer this question as stated (i.e., does WinBUGS actually process data?  Yes, of course), so I'm wondering whether some other question is intended.
@whuber: Sure! Depending on the data, the uncentered first PC may actually be very close to the mean vector. However, most PCA functions will do the centering by default, and the results do not change whether the mean centering is done outside the PCA function or inside or both outside and again inside.  And it may be quite hidden in the results, e.g. R's `prcomp` has the center but by default doesn't print it. I've also seen it reported it as the "$0^{th}$ component". So it may not be very obvious that it is actually done.
+1 thanks for the great answer
BTW, in reviewing this list I notice it focuses on *probability* exclusively.  Probability is not the same as statistics!
The help on perm.test says that the first two arguments are expected to be *integers*, and describes a somewhat surprising set of steps when they aren't ... and even then, it looks like it assumes the smallest gap in values is less than `tol` (is it?). It then links to a description of how the p-values are computed (in the help of another function in the same package). People attempting to answer might need to be aware of this. If anyone is inclined to look at code, it's in `perm.test.default` in the abovementioned package.
It is not always intractable, for example if your posterior is Gaussian you can sample from it. It depends on the posterior you are dealing with.
I wonder if a decision tree could be produced for deciding which type of model is best, given the answers the the questions you pose?
"*Suppose $Y$,$X$ and $Z$ are correlated random variables with $N(0,1)$.*" ... do you mean they are trivariate normal with standard normal margins? Or is the joint distribution unknown? When you put the subscript $t$ on, is there time-dependence either within or across series?
thank you very much, your response and the papers make your answer perfect.
I'd love to get an answer here, too!
+1 For giving the answer the OP needs, but I think he's more interested in guarantees/proofs that the coefficients are stable and that some kind of meaningful loss will function will allow him to recover the same coefficients.. or at least that's my interpretation of the question.
Nice answer. With respect to comparing canonical examples of discriminative vs generative classifiers (logistic regression and Gaussian naive Bayes respectively), I found this book chapter to be very accessible than the Ng: http://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf
If you're trying to predict dangerously low levels of antibodies, and you have continuous count data on antibodies, it would be better to model the continuous data as your outcome variable rather than the `acceptable` / `low` / `very low` binned version of that variable. (This is not binary BTW, so you would need ordinal logistic regression if you were to use this binned variable, despite the waste of information.) However, it's unclear to me whether you're really interested in simply predicting antibody count or whether you're trying to model longitudinal change, which is much more complex.
@ ssdecontrol: Thanks a lot for your explanation! Have you got an idea on how to visualize/plot the effects of such a regression? Boxplots for the three categories positive, negative and neutral for each of the subjects? Is that possible?
This is all basic material. I just didn't want to show the computations within this model without pointing out that the model is bad.
Sorry - I'm not a python user. So you will need someone else.
@Xi'an am not aware of a conjugate prior for for the dirichlet density. However, a google search turned up this post at MO which may be useful: [Conjugate prior of the Dirichlet distribution?](http://mathoverflow.net/questions/20399/conjugate-prior-of-the-dirichlet-distribution)
Binomial distribution with $n=3, k=2$ and $p= 0.05$ :-) $0.05^3$ - probability that all tests are significant; $0.05 \\cdot 0.05 \\cdot 0.95$ -- probability that first two are significant etc.
See the discussion [here](http://stats.stackexchange.com/q/67543/805) (which mostly covers it), and the last part of [this answer](http://stats.stackexchange.com/questions/76689/how-to-compare-the-mean-of-two-samples-whose-data-fits-exponential-distributions/76695#76695), which gives some links, and [this](http://stats.stackexchange.com/questions/94914/how-to-get-f-test-p-value/94945#94945) which gives some more discussion. Also see the comments under [this](http://stats.stackexchange.com/questions/55550/how-do-i-interpret-the-results-from-the-f-test-in-excel/55553#55553) answer.
I figured it out: A changepoint analysis would be good.
@probabilityislogic True, I only read about warm starts after I made the above comment. What do you think of this paper, which indicates warm starts are slower and sometimes less effective than simple cross validation? http://users.cis.fiu.edu/~lzhen001/activities/KDD2011Program/docs/p33.pdf
What difference would it make to include $0$ in or exclude $0$ from the support, given that this has zero probability in any case?
Could you please provide a reference that indicates SA has actually been used for this purpose?
@StéphaneLaurent I suppose that he meant the covariance of a DATA matrix. In this case, if we label $\\vec{a}_1=(a_{11},a_{12})^T$ and $\\vec{a}_2=(a_{21},a_{22})^T$, then $\\mathbf{A}=(\\vec{a}_1,\\vec{a}_2)^T$ and the sample variance-covariance matrix the OP's refereing to would have elements $\\mathbf{\\Sigma}_{ij}=\\text{Cov}(\\vec{a}_{i},\\vec{a_{j}})$.
Actually, the arima function in R will not fit your model (1). arima() does regression with ARIMA errors and your equation (1) is an ARMAX model.
I think you've messed up your parentheses; it seems to me it should be $\\min(x_i/i)$, not $\\min(x_i)/i$.
Thanks! There is a lot to look over here but it looks very helpful. Do you have a reference or a link for the equation of the "deviance residuals for the poisson model" or is it standard knowledge in a stat textbook?
It has a name ;) Thanks for the pointer. However I still left wondering "why" that particular selection of weights? i.e. why 4m and not 7m why not 2a or 7b etc.? Any theoretical grounds for this formula?
thanks a lot for your help, your answer is very appreciated!
About question 3, you can provide a minimal reproducible example, or a small sample of your data (10 records as plain text in your original post would be ok). Anyone could help provided it is known what the data looks like (e.g., are there external demographic variables that are of interest) and what question(s) you are trying to answer with your summary/inferential statistics.
Hmmm, this is not a flat prior at all...
Just a quick follow up - I did a similar thing to what you did, but I had access to the standard deviation of the data within each data point (ie. within each 5 min period). I divided by the smoothed STD and I got some pretty good results.
I think you are missing the point of a hypothesis test. It does not tell you whether there is a difference in the sample. In this case, there clearly is, because $8 \\neq 12$. Rather it tries to tell you something about the population from which the sample is drawn.
If I've well understood your question the interpretation you are looking for has nothing to do with your $\\beta$. You must first clearly define your dependent variable, that is, do you want to predict the probability that each nationality have of being first time visitors or do you want to predict the probability of which nationality will have longer vacation periods. Then, you can proceed with calculation and interpretation of your log odds.
Don't forget - the criteria by which estimators are judged isn't, typically, "fraction of time better than this other estimator", but more akin to "long run average performance".  So with your experiments, try calculating the average performance of J-S and LS over all your experiments instead.
Such distribution will have smaller variance. Try Hypergeometric.
With regard to determining power & effect size: you want a good chance of estimating an effect of an interesting size, given the errors.  A 'good chance' is relative to the bother & expense of doing the experiment. 'Interesting' is for *you* to decide, not a statistician.  You also need to estimate the error variance; if you've really no idea a pilot study would be advisable (as it often is anyway).
I don't know SPSS, but I wrote a [paper on ordinal logistic in SAS](http://www.nesug.org/Proceedings/nesug10/sa/sa03.pdf), and the output may be similar.
See http://stats.stackexchange.com/questions/79755/how-to-check-if-data-constitute-a-time-series-with-simple-moving-average-method As with that question, whether your data are time series is something that should be evident from the inclusion (or exclusion) of a time variable. If you are in doubt, you are not in a position to decide on appropriate analyses. On a different front, the distinction between a time series and a statistical sample makes no sense; a time series is also often a sample or thought of as such.
Hey Jacob, I want the "shape" of the output to have a certain form. That is, if I plot $i$ against the output, $o_i$, for each observation $i$, I want that plot to have a certain shape. So the idea was to take the entire set of the output from the NN, subtract that from the desired shape, and feed that back into the NN, so that the entire set of the output NN starts to approximate the desired shape.
Why do you want to do this? (For almost all statistical purposes, the choice of reference level makes no difference, nor does the number of coefficients that are positive)
Thanks for testing it
Matching reality trumps mindless model fitting. But you should be cautious that perhaps you want to modify what you think reality is. It sounds like, from your repeated comment questions, your whole point of asking was to remove the random intercept. You usually need a good 'reality' justification like the one Peter Flom mentioned. Perhaps you should add that to your question.
(+1) One discipline that requires a PhD is university professor.  (Of course there are exceptions, but they are few.)
Thanks, could you be more specific about "the RF is missing on some information"? Also, do you know of any literature on this topic of post-treatment?
And as such it doesn't seem too crazy for the RF/NN to shrink extreme values towards the mean, for better generalization...
I think the ramifications depend largely on the kernel you're using. I assume you are talking about a linear SVM and by "weights" you mean the actual support vectors? Since a linear SVM tries to approximate regions in the feature space with hypercubes instead of more or less arbitrary shapes, restricting the vector components to be positive means by my understanding that your hypercubes will all be in the first quadrant of your feature space, so to speak.
@Mark, I do not know what you are referring to when you say $S_{xy}$ or $S_{xx}$. If `y^bar` and `x^bar` refer to the estimate of the mean for either `no_claims` or `y` in your comment then they are not directly contained in the regression equation. What exactly is your goal in computing the regression equation (it doesn't seem you are particularly interested in interpreting the equation)?
Can you post a small piece of your data or a reproducible example, that we can understand the structure of your data? Or some things you already tried?!
+1, you are totally right about the stability -- $x_i^2$ can grow very fast. However, that depends on your implementation; keeping track of sum and sum of squares is less computationally intensive.
Thank you one more time!
Ah yes, I forgot that this F-test for u_i=0 only holds under homoskedasticity (try without the cluster option but either way the proper comparison test should be the Hausman test - with xtoverid you can do this also with cluster robust errors). The Hausman test is to see whether pooled OLS is consistent. The missing F value for the model is okay. In Stata the F(x,x) should be in light blue color - you can click on it and an explanation will appear as to why the F statistic is missing. I'm glad you are finding my answers useful :)
+1 for a plausable analysis and agreement with my answer to the previous question.  Would you agree with Michelle that the first model is better than the second.  I think a model with just income and sex would be better than either of the two above (verbal and status don't appear to add anything to explain y given that sex and income are included).
How do you decide whether the person's answer is 'right', once you have it?
Use a table of the normal distribution function $\\Phi(x)$ to figure out the solution to $\\Phi(x) = 0.75$. The (shortest of many possible) interval you seek is $(-x,x)$.
You should do several checks before going any further. First, if this is a small dataset ("small" will depend on other characteristics, but less than a dozen or so points might qualify), then bootstrapping is unlikely to be reliable. Second, have you tested your bootstrapping code? How are you performing this bootstrap? (In a now-deleted question you mis-characterized the bootstrap, so I'm suspicious.) Third, there may be a few outlying values of the response and independent variables in influential positions (further making the bootstrap unreliable): have you checked for these?
Beware of *any* dichtomizations.  The fact that the OP was tempted to make up 0.5 as a cutpoint means that you have not developed a proper utility function.
Statement or question?
No, there is no upper bound on the Poisson distribution.
Point is, that thest how is done was wrong and I was able to make enought corrections that it passes (by statitics) 20% of time. Each time that it fail test what happens like once in few years we said that we need to put roulette in balance. This test make inexperiance officials who had no idea what staistics is and even presenting them facts does not help much. All i currently know for sure is that i can distingush results from random oracle.
@Glen_b even understates the problem as there are also asymmetric distributions for which mean = median, such as some Poisson and binomial distributions.
Thanks for the suggestions.  Aniko is correct.
It's rather late to make this suggestion, but you might bear it in mind for your next experiment.  In order to assess whether blood pressure was high, medium or low, you must have actually measured it and got a result in mmHg.  You would get much better power from such an analysis using the actual blood pressures in mmHg than categorising them.  Additionally, categorising the blood pressures before doing the analysis, presumes you know how to categorise appropriately.  It is quite conceivable that for different treatments different categorisations would be appropriate.
@whuber, it could be sensible to only be interested in the association between $x_1, x_2$ insofar as it affects the regression coefficient estimates when they are both entered into the same regression model. This seems like an ad hoc method of diagnosing confounding. Although, if $y$ doesn't play any role at all, then of course the problem becomes much simpler.
@JonesTheAstronomer Yes, LLN is a theorem in probability theory: it follows from the Kolmogorov axioms of the definition of a probability space.
This seems inadequate because there is no mention of linearity. X and Y could have a strong quadratic relationship but have a correlation of zero.
@gung indeed, I would say that the concept of *ordinality* is ontologically prior to the concept of *quantity*. :)
This is a contextual multi-armed bandit problem.
I see no reason it can't be done.  What specific problems are you having?
Well, no: then your test data contributed to the optimization, and you end up with an optimistically biased performance estimate. Depending on how aggressively this optimization is done, I've seen the hit rate dropping for a proper validation from seeming > 95 %  to 60% (for a rather aggressive optimiztation scheme) or, for "mild dependence" from say, 95 % to 85% or so (that would be quite typical with our data).
@RussLenth Thanks for pointing. Added the same
You'll have to provide more background on what data you've collected. Can you clarify what you mean by "composites imputed by AMOS"? Is this some latent variable created from a handful of observed variables, such as survey responses and univariate or bivariate scores?
I am somewhat unclear on what you are asking. Let me see if I can rephrase; I ask you please to confirm or correct my understanding: (1) You wish to fit a piece-wise linear model to the original time series. (2) You want the turning points of the fitted model -- the points at which consecutive line segments meet -- to capture and reflect, in some sense, the "turning points" of the original series. (3) You propose to identify these "turning points" by comparing the behavior of the original series with that of a derived MA process, identifying "turning points" as points of intersection. Yes?
Something similar to that but includes more information rather than just the name of the test. We have some of these charts in urban and transportation modelling. They'll show a large table where they specify tests per type of problem. They also list caveats, expected time/duration, input and outputs, etc
I can't really answer your question, but this might be a place to start: http://en.wikipedia.org/wiki/Statistical_power
Can you please attempt to more explicitly define what you mean by the popularity of the content?
It is not a duplicate of a standard problem because of the inverse ordering .... so its not just a standard z-score job ! Why should I disregard useful results ? Would you care to give me a link or two to these duplicate threads ?  Not necessarily because I'm doubting you, just because there are thousands of threads on stackexchange and it would be rather nice if you could give me a pointer in the right direction seeing as you claim you know where all these duplicates are !
@NickCox Thanks for the clarification. I guess a different something extracts water from the tank.
Thanks! I thought something was fishy when I was leaving out the sum from 0 to 50. And I think I have more respect for the continuity correction now.
I have just read your answer - congratulations and thank you very much.
I'd appreciate a reason for the down-vote.
I wouldn't call this "mixed"; it sounds just like two-way analysis of variance. But the assumption is not that the data are normally distributed, rather that the error terms are. Styles differ, but informal graphical examination of the data might indicate e.g. that you should work with logarithm of glucose concentration. Similarly you could look at residuals from the anova and check for normality. Shapiro-Wilk [not Wilks, a different statistician] is a dedicated test for normality and superior to Kolmogorov-Smirnov if you felt a compulsion to include a formal test.
I was specifically wondering about residual analysis - requiring normality, constancy of variance and the rest of the linear regression assumptions. Likewise assuming iid observations. Then if you use a logit link, I believe one checks assumptions of logistic regression. I agree that the covariates need not be linear.
(+1) Good answer and welcome back, Zen. I was becoming concerned we might have lost you permanently given the length of your hiatus. That would have been a real shame!
What are i and j?
Assume that you perform two regressions for two hospitals. For one regression you have factors $x_1, x_2, x_4$ (and weights $\\beta_1, \\beta_2, \\beta_3$), for the second $x_1, x_3, x_4$ (weights $\\gamma_1, \\gamma_2, \\gamma_3$). Why don't you just fit two regressions with shared weights? Fit them jointly by introducing the constraints: $\\beta_1 = \\gamma_1$, $\\beta_3 = \\gamma_3$.
Yes exactly, I have Cohort 1 and Cohort 2, matched on a large number of variables (comorbidities), and the outcomes that I am interested in are binary, hence the logistic regression. Thanks for confirming my intuition on this.
This looks like a standard textbook problem. If this is for some subject, or otherwise for the purpose of your own study, would you mind adding the self-study tag, please?
Basic Markov chain theory is used to show that a particular sampling scheme will have a stationary distribution that is the desired joint distribution. For the simplest example, the vanilla Gibbs sampler simulates from full conditional distributions. The corresponding transition kernels, taken together, if they satisfy the conditions for convergence (often straightforward to show) can be readily shown to have the joint distribution as the stationary distribution. Similarly for Metropolis Hastings, and so on. It sounds like the lectures you're looking at don't explain how MCMC is a Markov chain
I can't say that I am familiar with this notation. Is this a programming language? Which one is it? What is vcd?