tttjjj/elt_bench huggingface dataset:
And about your second point: the reverse point can be made just as easily, IV supposes that the correlation with the error term is irrelevant.
+1 for teh maths...
Thanks for the tags Rob, I was unable to create the latex tag myself.
This is perfect. Exactly what I was after.
Just to clear something up, this additional term is just another covariate, and there is very little collinearity.
The answer has to depend what the response is. In this example, as it's presumably weekly sales, it might make sense to upweight it. That said, this seems unlikely to be the only quirk in the data. What about any week with public holidays? There is a fairly massive literature on this; conversely many software programs have all sorts of handles to deal with it.
Possible [duplicate](http://stats.stackexchange.com/questions/64788/interpreting-a-logistic-regression-model-with-multiple-predictors)
@Glen: A frequentist truly beholden to the LP (the weak version, equivalent to the Sufficiency Principle - the strong version is simply incompatible with the frequentist approach) *would* have to shun model checking. The ones who just admire it are glad when they can use it for the job of estimating the parameters of a specified model & still have more-or-less independent ancillaries - the residuals - left over for model checking any old how.
Thanks. I'll explore some of these models.
Hi, Thanks for your answers. So 0.000822 is computed as pt(-29603.3/7383, 18)*2
[1] 0.0008219592
So now that I know that pt(-29603.3/7383, 18)*2 computes 0.000822 how can I calculate the confidence interval ?
That is good, as I can see which one is trending but how do I a parameter whose value determine which is trending more as I have something around 1M time series sequences
Please check the [guidelines](http://stats.stackexchange.com/tags/self-study/info) for answering self-study questions (scroll down to "Answering...". If possible, could you try to guide/hint more on future questions (at least until the student has clearly figured it out -- or at least several days have passed -- at which point a fuller answer is good). You've left a little to do, which is commendable, but I think I'd normally lean a little harder toward leaving more for the student.
I don't understand what you're trying to do here. To estimate propensity scores you need to know whether each individual received the treatement or not, in which case you can simply count up how many received the treatment. Why do you want to aggregate propensity scores?
I want a quantitative number to outline how much does the measured value from a single location vary, how much does that number varying within a single organism (i.e. variation from the 5 repeats and from the 3 locations within the same organism), how much does the measured value vary between organisms and are these differences i.e. (location vs organism and organism vs another organism) any different. Essentially i'm hoping my data will outline that the variation from a single location is similar to that of the variation for any location from any organism.
This is explained in pp 2 of [this document](http://www2.isye.gatech.edu/~brani/isyebayes/bank/handout4.pdf).
Thanks, this makes sense. I am glad to see the asymptotic variance you posted. It is the same I found in other stats books - and the one I am using. However, I am a little worried that the Kappa variance traditionally referred in the remote sensing literature is very different; would you by the way also know the reason for that? The remote sensing version is the one reported by Congalton, page 106: http://www.scribd.com/doc/53393988/60/Kappa
What I'm looking for is a package that would take as an input a list of caret objects, and would then output the median, mean, or weighted mean average of their predictions.  More advanced functionality might include optimizing the weights through nested-cross validation.
Are there any reasons you can't average "confidence" in its ordinal format across the two tests and compare the difference in their numeric means?
Pearson's what? Two Pearsons are famous for their statistical innovations, so *lots* of tests and procedures are named for them.  Do you perhaps mean you are computing Pearson correlation coefficients?
@mbq Actually the code came from here: https://www.kaggle.com/wiki/LogarithmicLoss So I cannot change it.
Is the aim to predict the proportion of binary outcomes that are true for each input value?  This might be tricky for a Bayesian scheme as the correct method depends on the independence of the trials for each value of the input variable.  Can you give a bit more information about the nature of the problem?
@Srikant you're right, thanks.
Thank you. It took a little while to find a derivation, but it did help.
I've amended my answer above. But this is getting into programming specifics.
Could you give us a link and/or some context? I've not heard the term before, but maybe it is a kind of [stratification](http://en.wikipedia.org/wiki/Stratified_sampling)?
Any variable selection will cause a distortion in all statistical quantities.  But the issue may be also related to how arbitrary is the choice of the reference cell when looking at individual tests.  The global test ("chunk test") is invariant with respect to reference cell choice.
If the whole thing is viewed additively, with $\\ell = \\ln\\mathrm{L}$ being the log-odds ratio, then I guess $r\\ell$ would be the expected value for the weight of evidence from the unreliable source, if treated as a random variable (which is either $\\ell$ or 1, meaning the source is simply useless with a probability of $1-r$). A colleague drew up some reasoning based on the idea of a series of independent pieces of information, where we only observed a proportion $r$, leading to the factor $\\mathrm{L}^r$. I'd still like a more definitive answer, though…
@Christoph_J: Do you know this paper: http://www.indiana.edu/~kruschke/articles/KruschkeAJ2012.pdf - it is not so much about BMA but about convincing someone to use Bayesian methods in the first place - perhaps this is something that is helpful for you :-)
I'm not clear on how a continuous scoring rule can have spikes in the plots.  I'd appreciate some thoughts about that.
"statistical tests" won't tell you something *does* follow a power law distribution, since there could be distributions different from, but arbitrarily close to a power law.
Is the new value a "size" value like the sizes in the previous list?  In any case it doesn't seem to me like you have described the problem precisely enough for it to be amenable to mathematical analysis.
Oh, it's a numerical optimization! Okay, well, if you can solve it numerically, why don't you just try finding $\\beta$ by fixing $\\omega$ at various levels? Say $\\omega=0,0.1,\\dots,0.9,1$? It's a rudimentary grid search but it should give you a glimpse of what $\\beta(\\omega)$ looks like.
Have you tried adding skip-layer connections?  I usually run skip connections with a weight decay (aka ridge penalty).  If all the hidden -> output connections get shrunk to zero, then I tend to believe the interactions are weak sources of information for a given data set...
@whuber: The "only of" relates to the distributions for which the mid-points are the solution. You are right that there is indeed freedom in choosing the representative points, and there might be points which are not the conditional expectancy which will return the right answer. Anyhow, since the "only if" is immaterial to the answer, I will remove it.
If you're going to be calling R you will need to know enough to give it the right arguments (which will require understanding the help, which takes a little practice) and to be able to process what it returns. Which means you'll need to understand the data types and some of the syntax of R.
As we know, PCA is most efficient when the random variables are following Gaussian distribution, while in this case they are binary (discrete and non gaussian).
You should divide by $N$ of $N-1$ within the square root
No, its more of a case of me being lazy. Thanks for the help though
I think the OP is confusing confounding with interactions.
Thank you for the response and the interesting analysis!  This was very helpful.
Do you truly mean conservative in the sense that the actual type I rate is much less than the nominal? Or do you mean 'low power'?
I was actually going to discuss this same thing. I had a job evaluating CEP/ESP tools, such as Aurora, STREAM, and Esper once...
I guess that if testing accuracy is low there no good pick.
@ahfoss: I have added the code and improved the question
+1 From me - clear question with simple example and evidence of effort in trying to solve the problem. Not sure why this was down-voted?
This is also an interesting idea. Because some of the domains can be quite large, I'd probably group the data into $x \\times x$ km cells rather than $x^\\circ$ latitude-by-longitude.
You appear to have TWO forms of dependence here: By person and by category. So you probably need a multilevel model to account for both
@COOLSerdash: You could loop through a set of theoretical sample sizes like c(50, 100, 200, 500, 1000, 10000) and you could get an idea of how it impacts your statistic.
Well, as I said, I would run a simulation to see how the variance components look like for the pattern of regressors at both levels, as well as the relation with the outcomes. My gut feeling is that if you have outcomes that are strongly associated with particular clusters, and/or rare events, with whatever pattern, then I would imagine that you will have bad sampling distributions of the variance estimates for this sample size.
Hi, Chris. Though I and most others can guess at the meaning, it would be very helpful for you to set out clearly your notation.
Kappa is a measure of inter-rater reliability. How could such data be normally distributed? They are discrete, usually with a quite small number of categories.
@ttnphns: Oh boy, of course I simply mistyped my previous comment. In order to remove the possible confusion, I will now delete that comment and repeat what I said here with correct wording: "Reducing cardinality in this case means making your $N \\times N$ distance matrix smaller (decreasing $N$). Reducing dimensionality means making it lower rank, without changing $N$. PCA amounts to the **latter** and does not really help with the former goal".
I don't work with PGM but `PyMC` & `libpgm` come to mind considering Python. I haven't heard anything big and specific regarding C++. And then there is the Microsoft road with `Infer.NET`, that just based on the people working on it should be straight-up awesome...
I'm afraid confusion is bound to occur in matters of statistics. Your continuing efforts to clarify are most appreciated and unfortunately rare in our population of new users! We'll get to the bottom of this yet – I'll mull it over, keep an eye on answers, and offer my own if I have anything to add in the end.
Only problem is that god itself either eludes statistical reasoning or is statistically very unlikely to exist.
If each $w_n$ has a normal distribution with zero mean ($E[w_n]=0$ for all $n$) and the same variance ($\\text{Var}(w_n)=\\sigma^2$ for all $n$ - this is called _homoskedasticity_), then they are identically distributed. If they are independent, then $\\text{Cov}(w_i,w_j)=0$ when $i\\ne j$, i.e. they are not _autocorrelated_. Noise is tipically iid in experimental settings, when a measurement is repeated several times with the same instrument. If $A$ is not stochastic and $w_n$ is iid, then $y_n$ is iid because it's just a linear transformation of $w_n$.
The only problem with this as an example is that there is clear reverse causation.
This question seems to be *only* about how to do this in MATLAB. Thus, it would be off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)), but would be on-topic on [Stack Overflow](http://stackoverflow.com/). If you have a question about the substantive statistical issues involved here, please edit to clarify; if not, we can migrate this for you (*please don't cross-post, though*).
Bill Shneiderman has put together a nice history of treemaps with links to some relevant publications (http://www.cs.umd.edu/hcil/treemap-history/). Treemaps were  initially intended to display a multilevel hierarchy in a less cluttered way than dendrograms or trees could, and they were at first used for visualizing the contents of hard disks. Nowadays, treemaps are used for visualizing large phylogenetics trees (they show relationships between species), among other applications. For more examples, see the article by Shneiderman at http://www.perceptualedge.com/articles/b-eye/treemaps.pdf.
A few brief observations to get started: (**1**) Clearly if we start with $n$ tables, then after *exactly* $2n$ draws, there will be no tablets left; (**2**) The first time that you could possibly have $> 50\\%$ chance is after $n/2$ draws and this happens with probability $\\prod_{i=0}^{(n/2)-1} (1-\\frac{i}{n})$; (cont.)
Since you have labelled data it is simply a supervised classification problem and what you need is a binary classifier. Which classifier works "best" is a matter of personal taste and the specific problem. Definitely, I would try first some standard options (SVMs, decision trees etc.) before trying anything more exotic.
What's wrong with using a Standard Deviation?
@Andy I don't know. May be googling with the keyword "interval censored covariates" could give some hints.
Besides the things mentioned in the answers, your prediction intervals also won't have the right coverage.
What is the substantive question you are trying to answer?
Okay, thanks. The situation you describe is one of the possibilities covered in my answer (which is that in the worst case you need 95% of the population).
@whuber Yes, it is! My skept-o-meter says that the details of multiple stories may not be quite right, though. Perhaps both of them solved open problems, but maybe one of them knew the problem was open and the other mistook it for a homework problem. The stories would match in terms of "naive young student solves open problem, not having realized how hard it really was", but when the story became legend and someone told it as "some student ...", someone else chimed in, "oh, that was Dantzig", or "oh, that sounds like Milnor" and the stories converged.
Maybe [this post](http://stats.stackexchange.com/a/67508/21054), [this one](http://stats.stackexchange.com/a/95333/21054) or [this](http://stats.stackexchange.com/a/62917/21054) can help you applying the delta-method.
@gmacfarlane, of course there are tests for non-nested models but the likelihood ratio test is a specific test, and it doesn't apply unless one model is a submodel of the other - see [here](http://en.wikipedia.org/wiki/Likelihood-ratio_test) for more info. If you have a specific test in mind (i.e. if you care to educate us on what a "non-nested likelihood ratio test" is) it might help if you gave some more details. I skimmed the link you posted and didn't see anything like that (of course, I don't have the 2-3 hours required to actually look carefully at it).
I'm not sure I understand your commnet. If b1 and b3 are correlated, why does it matter? I  mean, their joint distribution has to characterized with some correlation parameter, but so what? I do have their marginal distributions. 2. I didn't understand you mention about the  the law of large numbers. Could you expand on it? Finally, are you sugesting that I should add b1 + b3 in the main loop? And I just need to use a vague prior to this new parameter? Thank you!
Can you explain how [dynamic programming](http://en.wikipedia.org/wiki/Dynamic_programming) enters into it? That seems incorrect to me.
Thank you for the hints. The mentioned "best subsets regression" seems to do what I would like to do. Select the attributes which are in some way more meaningful than other attributes for the clustering. Jain [link](http://www.cs.rutgers.edu/~mlittman/courses/lightai03/jain99data.pdf) describe this step as _feature selection_, but how to do this is a bit vague. So it seems that is more of an iterative process and some kind of art to find the good attributes than a determined process where I have the best attributes in the end.
I was thinking of the your answer, but I just wanted to confirm that was correct. In my text they it mentions that they regress $y_t$ on $x_{t1}, x_{t2}$, and $t$, and get the equation:$$\\hat y_t=\\hat\\beta_0+\\hat \\beta_1 x_{t1}+\\hat \\beta_2  x_{t2}+\\hat \\beta_3 t$$ So I would assume you mean the constant $\\beta_0$?
Have a look at [this](http://stats.stackexchange.com/questions/29271/interpreting-residual-diagnostic-plots-for-glm-models) and the links in the answer. It is much more comprehensive than what I was going to say here. Depending on the problem, you need to revised your model, sometimes removing a variable, sometimes transforming it. I cannot provide a single remedy, it is depends to the problem.
I think your terminology is problematic. Groups tend to refer to subsets of cases, I have changed your terminology in your question, however if I have misconstrued feel free to change back.
Hey Glen! Does my edit clarify things?
(+1) Thanks a ton for your help. I managed to make a violin plot and also think that I have reshaped the data correctly. Am still having a bit of trouble with stratification and other analyses though. I posted a new question: http://stats.stackexchange.com/questions/66030/comparing-weights-versus-time-graphically-using-reshape-vioplot-and-ggplot2
(A) for classification the output of the function we want to learn $f(\\cdot)$ are class labels, (B) you try to build a model to learn the mapping $S\\mapsto O$ and then use it to predict an optimal output for a new S.
@ZKhan You are welcome! If this answer was useful, you may accept it.
I'd think your predict model should be selected based on the observed data and use the model to predict the missing values during multiple imputation as well as the pooling analysis.
As @Alecos points out, you're using "truncated" idiosyncratically. "Censored" is the usual term.
"Puzzled" is the best start to learn and really understand anything - at least from my personal experience!
Andrei I don't think it's usually anything so formal; for most people the Weibull is a convenient ... well, *convention*, one that happens to often be a reasonable model. In some situations there may be a circumstance that leads us to expect the Weibull will approximately hold (I believe there are a number of characterizations of the Weibull), but for most people modelling survival times, those justifications are no more formally considered than when people assume normality to apply a t-test (they don't usually work out whether there are many small, not-too-dependent components being added).
I see what you mean about nonlinearity... I get correlation coefficients of -0.14 (numbers 1-12), -0.06 (numbers 13-28) and -0.05 (29,30). It makes sense to assume that all dates chosen will include the month, so I'll focus on the numbers 1-12, which do indeed seem to be over-represented, if I understand this (anti-)correlation correctly.
@user1275515 - Glen's answer is good. The distribution of the length of time between two events in a Poisson process is exponential. It's worth taking the time to figure this out fully before worrying about the parameter values.
There is a package in R called R-INLA (Integrated Nested Laplace Approximation) which is increasingly being used to model latent Gaussian Random Fields (with covariates). See a nice paper [here](http://onlinelibrary.wiley.com/doi/10.1111/2041-210x.12017/pdf) which has some R code in the supplement to get going. There are also several other papers on the INLA webpage [here](http://www.r-inla.org/) several of which have R code. INLA is is very computationally fast, but it may be a bit black-boxy for some. Regardless, it is the state of the art as far as fitting latent Gaussian models.
@Matt: I added my response as an edit above.
Notice "Therefore groups that I may be comparing may have different distributions".  I don't know if the distributions are different or not.  That is the point.  Between the two groups I may have two normal distributions, one non normal and one normal, or both non normal.
Deuterium - rather than make such extensive edits to the answer, perhaps suggest to lebatsnok those parts be removed in the light of your edits in response to them.
How to implement the MLE method from referenced literature into my code?
There's also an R package zipfR http://cran.r-project.org/web/packages/zipfR/index.html
I haven't tried it though.
Interesting question; I would say it is in general an error to do CV this way since CV assumes there is no information in the order of objects,  but I'll leave the field for local ts experts.
I think you will get a better shot at answers by migrating this question to SO. But let me ask you: is there anything the matter with just fielding an exception thrown by `solve`? That actually could be one of the most efficient (and relevant) ways to check for problems.
Q1: population avg effects vs. individual effects equivalent to fixed vs. random effects? A1: NO. fixed effects are individual level parameter estimates, random effects (or random intercepts) is the random variable describing distribution of individual intercepts in the population (due to infinite unobserved variables & CLT usu. a normal RV). Q2: If normality of random effects is true and we correctly specify covariance structure, aren't individual level and population level effects the same? A2: Only when the mean model is true and under heteroscedasticity. Otherwise, they're usu different.
Perhaps your question is answered at http://stats.stackexchange.com/questions/9131?  If not, what else would you be looking for?
@MichaelChernick: If you look at the `R` call the OP is making and the associated output he provides, you will see that he is, indeed, using the lasso. As I'm sure you know, a small tweak of the lars algorithm yields the lasso regularization path.
If this is a homework question please add the self-study tag.
Thanks for the term **multivariate optimization**
Your last paragraph does not really address your original question: because women tend to be shorter than men, it seems of little value to test whether married women are shorter than married men.  What might be more meaningful for your research is to test whether the taller women are married to taller men and shorter women to shorter men.
yup, igraph is the way to go for visualization. You have to compute the transition probabilities beforehand on your own. In general, I recommed to take a look at [Markov Chains](http://en.wikipedia.org/wiki/Markov_chain)
@kwak: Glancing at the GPUtools help file, it seems that `useSingle=TRUE` seems to be the default in the functions.  Am I missing something here?
If $u$ is continuous, $\\{u|v,S\\}$ means the conditional density of $u$ given the values of $v$ and $S$. (Usually, as I have done you'd put a comma between them though.)
Perhaps write out the prior and likelihood you are using? Just giving us the names does not mean we are syncing on the letters/quantities used. Many texts/resources differ wildly from each other.
I thought I had heard that the set membership values used in Fuzzy Set theory weren't probabilities (At least, that's what I remember of Zadeh's statement regarding it).
If the code doesn't work with your data, your data isn't what I assumed it to be. That's why on Stack Overflow (where this would be more on-topic) we ask for a reproducible example, e.g., the output of `dput(DF)`.
A cophenetic matrix would be a distance matrix wherein original pairwise distances between the objects are replaced by the computed distances between their clusters at the time of these clusters' merge. Now, most hierarchical clustering programs do not construct such a matrix (no need for) and they plot dendrogram out of another source (such as agglomeration history protocol).
@NickCox: My guess is that he has exact dates of the `wat` measurements, but may be using reported (NOAA) monthly rain and temperature values, so he's aggregating to the most granular measurement. Any clarification, user1868064?