tttjjj/elt_bench huggingface dataset:
Could anyone please answer my latter question regarding my thoughts on how to calculate factor loadings, about the use of principal component method for estimation and about my thoughts regarding adjustment of the missing values?
Thank you so much, Glen_b!
Thanks for your reply. I need to assess the performance of each algorithm based on their results for 1 week. Every minute of simulation I get the mean+stdev of the response time spent for all requests during that minute. My concern is that I need a way to infer/compute the overall performance from the collection of 1 minute means+stdevs. Since 1 minute can have 10 requests and the next 100, they don't have the same relevance in the overall behavior. My reasoning is that I should account for that with some kind of weighting, hence my idea of using a weighted mean/median.
I don't think it takes 10 weeks of exceeding the mean+1.645 sd threshold to make an epidemic call ... more like 2 or 3.
@Tomek Tarczynski: I have used it a small amount. There is a package in Matlab (I am sure there is one or more in R), which I have used. It also provides a sparse PCA, which I was more interested in. I admit I only skimmed the paper. ;)
+1 to this answer, but also adding that even if the most recent 3 months are the only ones that correlate to the answer, there could also be factors like the same month of the year in the last 5 previous years that are useful inputs (e.g. if Decembers tend to look like previous Decembers, typically).
Can you first give some more details of your context? Give that by editing your original post!
Can you explain what is meant by: i) you "found one breakpoint?" How did you find it? ii) What do you mean by "same decreasing slope?" Do you mean both slopes being negative? Or both slopes being negative and with similar magnitude?
@gappy, sure. Regards.
AIC generally yields less underfitting than BIC.
BTW, my answer has another thing going for it: in your original post, you mention that the reason you don't want to simply use the sample standard deviation, calculated directly from the Z values, is that in that case, you would, in your own words, "neglect the fact that there are known measurement uncertainties". My answer (well, really, the obscure textbook answer, which I'm simply sharing with you) uses the known measurement uncertainties, exactly as you asked for. It's just that it uses the information in more places (mean result as well as standard deviation) than you had been expecting.
Other people mention statistical procedures which may be helpful, but I would first ask whether you have any theory about the strength and shape of the relationship between variables. How big is your sample? Do you have reasons to avoid complex models?
You can find this result in the wikipedia article [Conjugate prior](http://en.wikipedia.org/wiki/Conjugate_prior). I hope this helps.
$3x^3 + \\frac{3}{4} x^2$ is a possible alternative.
So you want to test if the combination of two medicines is better than the individual ones?
Many thanks Nick.  I'll look into that.  Basic problem: hundreds of dummy variables. I was wondering if it would be valid to group these using k-means or similar...
I am not sure that it is meaningful to speak about significant loadings in a non-specific sense: before extraction all variable variance is fully distributed across factors, so what would it mean to be significantly loaded? If you choose fixed thresholds than these would have to be realtive to the number of factors. The more interesting question is: why is it useful in your view to have this threshold? What is your motive to classify variables in this way?
+1. Especially for the suggestion about modeling the spikes. The data look a lot like daily retail sales with promotions (upward spikes) and availability problems, also known as "Out of Stocks" (the downward spikes).
@peuhp Yes, I meant to say that the $x_{i}$ are assumed to be i.i.d.
Where exactly do you need help with these questions?  Please see our [help] for guidance on how to ask good, answerable questions.
For this analysis to make sense, the scales for each of the 6 variables would have to be the same.  (Recall that ANOVA compares means.)  If you used the same scale for each variable, then some form of repeated measures ANOVA may be appropriate.  How many managers and facilities do you have?
Could you perhaps present an example of what you mean?
This question appears to be off-topic because it is asking for code to use statistical software.
@Glen_b The choice of plotting the reciprocal actually works pretty well here and in any situation where the data are already rates or ratios it has a natural interpretation.
@Peter Interesting thought; but no: make a plot of the expected mean as a function of $t$ to see what's going on.  (Use a logarithmic scale for $t$.)  There are deep conceptual issues involved here: randomness is not arbitrariness, nor is all randomness describable with a (fixed) probability distribution.
If, as you suggest, the variance assumption is okay, then it's probably not going to lead to any problems at all just dealing with OLS regression.
@NeilG, one reason to use the probit model is that it gives a convenient way of modeling multivariate binary data (e.g. with a mixed model) as thresholded normals. In that case, the correlation matrix of the underlying variables is statistically indentifiable, whereas it is not in the logistic case. There's a bit longer discussion [here](http://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models/30944#30944).
My worry is primarily with the interpretation of the multinomial logit model in this case.
...and regression coefficient beta is directly related to partial correlation (see the link under the Venn diagram in the linked answer).
But that's the same as ancova?!
Missings that really mean 0 (or something else) can be recoded as such, although that might have implications for what model is sensible. Missings that are recoded arbitrarily will usually lead to nonsensical results with any model. Otherwise the only option is imputation.
Wow I can't believe it was that simple. Thanks... I checked this with other models that I knew the likelihood and it is correct.
@Tim: I distinguish between the estimated pFDR and the true pFDR. The true pFDR of any rejection region will surely depend on the sampling distribution.
One way to interpret the OP question "does the posterior need to be a proper distribution?" is to ask if it is mathematically possible to start with a proper prior and end with an improper posterior. Minka's answer gives an explicit example in which it happens. I tried to complement it with my answer and point out that this can only happen inside a set of zero prior predictive probability.
The problem itself usually is not rotation invariant, because each of the variables is recorded with a conventional unit of measurement appropriate to it.  E.g, $a_1$ might be in feet, $a_2$ in microns, $a_3$ in liters, etc.  Even when all units are the same, if the variables measure different kinds of things, the amounts by which they vary will likely differ in ways characteristic of those variables: once again, this is not rotation invariant.  Therefore you should abandon rotation invariance as a guiding principle or consideration.
I agree with the first half of that but I would point out for the second half you can run a simulation as many times as you like, changing assumptions until you find something that reasonably approximates reality. How difficult that is can depend on the complexity of what you're trying to model.
Perhaps the test you are performing is a significance test of all variables in model (SNP + covariates)? Make sure the test is only testing whether the coefficient on the SNP variable is 0.
(+1) And, welcome to 1000 reputation. Cheers.
Although discussed in the context of logarithms instead of roots, you may be interested in reading these threads: [How small a quantity should be added to x to avoid taking the log of 0?](http://stats.stackexchange.com/q/30728/7290) & [How should I transform non-negative data including 0's?](http://stats.stackexchange.com/q/1444/7290)
Are you saying that the "all stores are similar" assumption implies that their time series means are the same? Otherwise, two identical stores can still have very different means due to location.
@DouglasZare, that seems like the answer. I wonder if you might want to make it official...
Hi Matt. I posted an addition with a link. It seems like these techniques are used for dimension reduction. Do you happen to know how to use discrete wavelet transform in R to do dimension reduction?
BTW, as Marc Kery says, no model is Bayesian :-) Bayesian is only the way of inference.
It would help if you answered the questions I asked for your preceding question... (and please register your account so that you could vote and accept @Jeromy's earlier answer.)
I believe that your series can jump all over the place, but have a (near) constant mean. That's why people detrend or difference time series.
One rule of thumb: use enough points that it looks like a smooth curve when you plot it.
Thanks for the answer, and thanks for noticing that I bolded "Econometrics" in my question! It is indeed under an econometrics context that im learning the CLT.
That seems like a logical result to me. If you arbitrarily increase the bootstrap re-sample size, the estimates will be less variable. I presume such an arbitrary increasing of the bootstrap re-sample size is totally inappropriate, so what is your motivation for doing it at all?
You should add your code and the two plots.
I suppose the take-away is that there is no convention and, most importantly, no one will strongly object if I use one definition or the other.
What do you mean with overlapping region? Do you mean the area that is below both density curves?
No, very little has been published. Quinlan has said that there are some occasions that ensembling a tree might make it worse (Quinlan, J. (1996). Bagging, boosting, and C4.5). I didn't think that would happen but it does. Someone (Stan Young I think) had the opinion that ensembles don't help trees much when the predictors are uncorrelated (Although I've never tested that).
Ok thank you for answering :)
I think answers like these belong in the comment section. thanks for being nice @caracal.
Why wouldn't it be?  Sounds fine to me.

And welcome to the site. I deleted your sig line (the site adds it automatically) and edited your post
Very interesting, an extremely subtle question.  I think the point he is getting at @MichaelMayer is that although $|X-\\mu|$ is numerically $|\\mu-X|$ the distribution of $X|\\mu$ is not the same as $\\mu|X$.  My feeling is this either leads us into Bayesian statistics, where it will all fall out fine once you specify priors etc., or into those tortured lawyer-esque frequentist statements about exactly what a confidence intervals is.
I've been meaning to look into this as well. Never really understood this claim. Though my interest lies more with the use of random forests and boosted regression trees in identifying interactions.
I see, that should have made the thing clear from the beginning. Thank you all!

I suppose there is no easy way to get the PDF of the underlying distribution $p(x_0)$, where $x_0$ corresponds to the point which has $P(x_0) = D$?
R122, I asked that question as a hint to you: you needn't reproduce the text in comments! Now it's time for you to apply the definitions you quoted to this example.
I've edited your answer to include mathematical text formatting (Mathjax, which is basically LaTeX). Please double-check that I have not inadvertently introduced any errors.
This sounds like a classification question. Have you looked into something like decision trees?
Thanks to all who helped me to solve my problem.
@TonyHaenn - I've added some stuff into the answer.
nice, I didn't know "unclass"  ;-)
Biostatistics also makes unique contributions to statistical theory, and the links between study design and inference. Equivalence tests are an example of the former, and counterfactual causal estimation *a la* James Robins, Sander Greenland, etc. (within epi/biostats) is an example of the former.
What is the "ID3 algorithm"? If it requires you to discretize continuous variables, it may be better to use some other method. Certainly there are classification tree methods (from your tag) that do not require binning.
Any takers? I was hoping this would be a straightforward question for statistically-sharp folk on here.
@cardinal, thanks. I had thought of that --
weight( 32 bits ) ~ Normal( 16 +- 2.8 ) --
but then for N = a million thought that E normal^N would be inaccurate;
in fact I get 2.2 +- .5 vs. Monte-Carlo ~ 3.5 bits, dunno. A direct, intuitive way would be nice.
@JEquihua: c-means is continuous rather than hard, so each point has a value [0, 1] for each cluster. So if a particular point has its highest value for your center cluster, you can look for its second-highest value and put it in that cluster. The problem is that near the center of your forbidden cluster all of the other cluster values may be very small and nearly the same.
Great point about the need to consider the population! That's one of the big ways people can get lazy about big data.
About Barr et al., it is important to keep in mind that their conclusions about omitting random intercepts vs. random slopes do apply to the particular experimental design they studied (the "fully crossed" design), but do *not* necessarily apply in general! Which components matter most depends on the proportion of predictor variance that varies within- and between-units. If all predictor variance is between, it is the intercept that matters. If data are highly unbalanced and many missing values, both will matter. So the correct conclusion is more complicated than "Omit intercepts, not slopes."
"The coefficients [...] can be interpreted as odds ratios." Do you mean the exponentiated coefficients in the equation $\\frac{P(Y_i = 1 | X_i)}{P(Y_i = 0| X_i)} = e^{\\beta_0} \\cdot e^{\\beta_1 X_{i1}} \\cdot ... \\cdot e^{\\beta_p X_{ip}}$?
@ chl. OK. Tx. Relevance to the question.
@Cam.Davidson.Pilon: I think *the posterior probability the coefficient is greater than 0 is...* is a better choice than anything to do with p-values
@ttnphns I disagree with (2). That is Lloyds algorithm, not generic k-means. K-means in general means minimizing the sum-of-squares-partitions objective. What you described is the generic expect-maximize (EM) pattern; and Lloyds is the EM pattern for least-squares models.
Recognizing the geometric distribution of # of boys in the family is the key step to this problem. Try: `mean(rgeom(10000, 0.5))`
@GraemeWalsh Embrechts Quantitative Risk Management and Härdles QF book. Embrechts sometimes summarizes the ideas shortly, but it is not sufficient for me.
In the end I don't care of preference but only what they really eat. If they buy stuff they like less but do because no better product around then this is a fact I must accept. Local diversity is still interesting though
I would disagree with the characterization of p-values suggested by the first formula. The correct sense of "at least as extreme" in the Neyman-Pearson theory is in terms of *relative likelihood* and not in terms of the usual ordering of the reals (as indicated in the formula).  The two are equivalent in many standard testing situations but differ sharply when the sampling distribution is bimodal. This distinction will therefore resolve the question satisfactorily, I think.
This is a pretty broad question that has been discussed on this site quite often - have you done a search of all questions that use the tag 'model-selection'? The accepted answer (given by gung) in this thread http://stats.stackexchange.com/questions/22902/model-evaluation-and-comparison-for-selecting-the-best-model may be a good starting point
Without any kind of a definition for what $p$ even is?
@Peter Best kind of depends on a lot of things - what sort of treatment one seeks, for example. I also prefer to read many sources rather than one - including papers, talks, notes as well as books. But the classic references of Huber (1981) and Hampel et al (1986) are still pretty relevant (see the wikipedia page on Robust statistics for complete references and a number of other good ones).
I was suggesting that "automatic identification" of a specific distributional form in exactly the sense of your link is usually a bad idea.  I even explain why in my answer. Explained another way, it's a bad idea for pretty much the same reason that in regression throwing a hundred sets of three predictors at a response and choosing the most significant set of three is usually a bad idea.
Explanation of boxplots is usually poor, a situation that perpetuates itself in several literatures. In particular, the Tukey convention that points are shown individually if and when more than 1.5 IQR from the nearer quartile is (a) not universal (b) not something that can be assumed obvious. http://www.stata-journal.com/sjpdf.html?articlenum=gr0039 is one reference discussing variations, which gives others.
Because there are myriad ways to express PDFs, a good solution should aim for a representation that is suitable for your purpose: what, then, is the objective of this exercise?
Wow, great resource. Thanks for putting it together and sharing!
@rolando2: This is a recently migrated post. If you were, by chance, the one that downvoted it, you might reconsider since the OP has not yet visited here and may not even (yet) be aware of the existence of this particular SE site. Also, if you have a potential duplicate in mind, it would be useful to provide a link and/or flag it appropriately. Cheers :)
Thanks! This exactly what I needed
This question is far too general to be replied in a relatively short answer. Maybe it would help you to watch Hinton's talk http://videolectures.net/jul09_hinton_deeplearn/
This sounds less like a programing question and more like you need help on choosing an appropriate statistical method for small sample data. This type of question would probably be better on [stats.se]
@Amateur so your question is about the interpretation of a regression line and has nothing to do with the mixed model ?
+1 A little cynical perhaps, but I'm sure many of us privately provide similar assessments when we encounter similarly vague statements in papers.
The individual y's are Poisson. $\\hat y's$ are some complicated function of the set of $y$'s and the $x$s. The difference is not normal. The point of the residual plot is if the Poisson means aren't too small, the Poissons will be approximately normal. But the observations with small Poisson means won't be very well approximated by normals - and neither will the residuals. So you can't necessarily expect the QQ plot to match well everywhere, especially in the lower tail.
Compare - I meant comparing the Arch and Garch coefficients for example. 10y+ is the long-term overall norm and to see what different characteristic this particular short period may show in the results. It's not a like-for-like comparison, but it's more about any deviation/anomalies from the norm in that particular period.
I've run some simulations on this to verify the answer and I encountered a few problems. It appears that choosing $w_i=N_iN/(N-1)$ does not give the correct result (it is scaled with about $k$). Plugging this weight into your formula for $E(\\hat{\\sigma^2})$, gives $E[\\hat{\\sigma^2}]=\\sigma^2(\\frac{kN}{N-1}-1)$. Choosing $w_i=\\frac{N_iN}{N-N_i}\\frac{1}{k}$ solves this, but the resulting estimator underestimates $\\sigma^2$ when the number of groups is small. I wonder if the expression for $E[\\hat{\\sigma}^2]$ is correct; I've tried the calculations but I have a hard time following them.
CA is in essence an exploratory approach, so I'm afraid that whether cluster results make sense is a matter of your knowledge in the field of the study. However, if your question is about *characterizing* clusters properties afterwards, then there are a lot of possible way to describe the results using standard summary statistics. If so, it might be good to add a concrete example or reformulate your question in that direction.
Imagine three people, A, B, and C. For all three pairs (AB, AC, BC) to be friends, you need three friendships, each with probability $p$.
A common analogy is shooting at a target with a centre. (Assume a benign version in which no one gets hurt, say the sport of archery.) If you hit the centre on average, that's accurate. If your hits are tightly clustered, that's precise. You can have precision with or without accuracy. Now take away the analogy: in real science, we can't usually identify the target....
Hehe, well spotted Sir! :) If I omitted any ventures into the land of creative branding it was merely because I felt compelled to support the resourcefulness and dedication of your effort and ideas rather than to concoct vain labels. Since you insist on naming the mean of the integral, beware that what you consider the "mean of the integral" is a simple mean of your deltas. And as such, your outliers are simply "deviations from the mean", or possibly deviations from the local mean. I don't quite see the advantage of thinking in integrals, unless you don't have enough sampling points.
This answer was spot-on. Thank you!
Why are you using a random forest instead of a logistic regression? (The feature space is quite low dimensional.)
"_...i try to predict the **next** 10 days_", so you would need the date index.
Jan Beran's book http://www.amazon.com/Statistics-Musicology-Interdisciplinary-Jan-Beran/dp/1584882190/ seems a natural starting place.
Upvoted not just for the good answer but also for the first paragraph.
Sorry, I understood you were referring to the opposite situation. You can check the significance of the linear trend in a model for the differenced series:
$y_t-y_{t-1} = \\Delta y_t = c + \\delta t + \\epsilon_t$.
You can also apply the unit root test to the differenced series $\\Delta y_t$ to see if there is a second unit root. You may stick to the model with intercept (unless a graphic of the differenced series shows an exponential pattern).
Do you mean that the way I've written the model makes the normal distributions priors? Because that's not what I'm intending to do. My data consists of M plots and at each plot I have a value that is measured over time (P=M*#time-points). I want to fit a damped oscillator (the Yhat line) to each plot; I suspect that the damped oscillator parameters (amplitude, damping coefficient, etc.) are similar from plot to plot. So I want to have hyper-parameters on the damped oscillator parameters that are normally distributed. I am really interested in the mean and variance of these hyper-parameters.
Your proposed model misses some important aspects.  This would be clear if you were to redraw the second figure (the scatterplot) in a way that distinguishes time, because the upper envelope occurs between 160 and 290 days and the lower envelope at other times.  Evidently $P$ depends on time $t$ and so does $T$.  Separate out these dependencies, writing $P(t) = g(t) + \\delta(t)$ and $T(t) = f(t) + \\epsilon(t)$.  The task is to identify $f$ and $g$ under the supposition that $\\delta$ depends on $\\epsilon$.
How would you apply lme  here?
Do the estimates of a, b or DDREF have a standard error associated with them in the studies?
Could you please explain what the "Original curve" in the bottom plot represents?
"Virtually identical" is the basic idea, Peter.  Even when two questions have been phrased differently, if they really address the same questions and answers to one are likely to be answers to the other, we can merge them.  This causes one of the questions to vanish, but all responses are preserved and united under one thread.
Yes, I could compute the Mahalanobis distance of each observation from the mean of the data. But the observations with the greatest distance from the mean are not necessarily multivariate outliers. Think of a bivariate scatterplot with high correlation. An outlier can be outside the main cloud of points but not that far from the mean.