tttjjj/elt_bench huggingface dataset:
(1) True, I corrected my answer. (2) I tried to help you out, but I can't read papers for you. If you want to understand, as you title states, you need to be able to read for yourself. If you get stuck and have a concrete question, ask, but first you have to do the reading yourself.
For me, spatial dependence is a characteristic of the data generating process (DGP), see @Emre, and spatial correlation is a characteristic of the pattern/configuration of your observations (empirical). So, it is expected that a spatially dependent DGP will result in spatially correlated values.
I think maybe I should just use a Cornish-Fisher expansion instead...
@Abe corrected thank you
Cool, thanks Greg - I'll run the eigen() function to make sure my variance-covariance matrices don't violate the positive-definite limit.
@Thomas, no, it's the diagonal matrix as you've specified it in your initial post, but where the $\\pi_i$ are replaced by the estimates $\\hat{\\pi}_i$, i.e., the estimated probability that the $i$th response is 1 under the model.
(1) I think you meant to write that the *sample*, not the *population* mean, is 112 with 95% *probability*, not *confidence.*
(2) Your point is well taken--it could apply to any question--but isn't it stated a little extremely? First, the question doesn't ask for an inference about the population mean: we are told it is $\\$$200. Thus, conditional on the assumptions, we can estimate the population mean with certainty! Second, even if we were asked to estimate the population mean from the sample, there's still some trivial information we could offer (e.g., it doesn't exceed $10^11 per annum).
The solution is to not use relative error in this case.
If you want an answer to include **R** code, it would probably help if you include your current **R** code. Just the `lmer()` call should be enough.
Regression is used as it tests for one form of heteroscedasticity. In other words, it tests if error variance is a function of the regressors which if true would make the error heteroscedastic. Consider the OLS residuals. An estimate of error variance is: $\\sum_i(e_i - \\bar{e_i})^2 / n$. But, in OLS, $\\bar{e_i} = 0$. Thus, an estimate of error variance from each observation is the square of the error residual.
"whuber" agreed !
@EpiGrad: I am currently bound to R so am not really sure how much of this is really far from impossible :) Are you aware of any material that discusses this using R?
its for class we had to indicate possible significant from our lm then create another lm with just the two variables which I did and I did a logit and it does indicate that sex and income are significant.
Can you specify *your* question, as distinct from the homework question you were given? What is it you are confused about? Be aware that our policy is **not** to answer your homework questions for you, but to provide hints to help you think through where you're stuck. You may want to read our [FAQ](http://stats.stackexchange.com/faq).
Fair enough. But, you raise three questions at the end of your reply, and I'm afraid comments provide too little space for serious discussion.
@chtfn There seems to be a problem with variable labels. A quick look at the source code indicates that this error message comes from `insert_absorb()` which tries to extract pair of treatments. You can perhaps try to change the separator you used for coding levels of your interaction term? Without a working example, it's hard to tell what happened.
At least for me the key understanding is that I can't just say "I have X, hence (1/2*X + 2X)/2 > X" - the *total average* chance is 50/50, but for any specific X, the expected chances aren't 50/50 anymore; and the larger X, the lower chance of having 2*X in the other envelope (for positive finite distributions); so integrating over the possible X'es  `sum(p(X) * (1/2X*f(X) + 2X(1-f(X)) ) = X`, where f(X) is the likelihood of the first envelope being larger, given any particular X.
I understand your concern about whether this problem belongs on this site.  I'm sure there are alternate algorithms I could try, but they require more elaborate programming and quite possibly introduce their own unknowns, such as task dispatching latencies.  If I understood more stats, I might have been able to abstract the problem and not even mentioned computers.  Perhaps what I'm asking is something along the line of "what is an average?"  What I'm hoping for is that someone will be able to identify a generic statistical problem behind what I've described.
Looks good, Chris. Also, do some simple simulations to see it working. For example, find the mean of a Gamma rv sampling from a standard lognormal rv. It takes only a bunch of lines of R code.
I think Feinerer's dissertation is the document that has helped me most so far. Thanks!
Is it a case for a two-way ANOVA with repeated measures in one factor (because there are 2 factors influencing the density [cells/ml^2]: treatment and time)? http://graphpad.com/support/faqid/1107/
I don't have any recommendations, because by the visualizations there is only one cluster in the data. The data might be just a "chunk"?
I agree that 0.5 looks smooth and yet shows a reasonable trend. It feels un-statistical to talk about "looks", but I don't know enough to go beyond that. What does the detrended data (i.e. residual after subtracting your loess trend) look like?
@nico Cross-correlation tests require joint normalitywhich requires normalityforeach series.Normality requires (among other things!) independence for each and every observation in the series. Your Time series data the data is (probably) auto-correlated violating the assumption of independence.You might start with http://en.wikipedia.org/wiki/Udny_Yule and "Why do we get nonsense correlation between time sries".You might also read http://autobox.com/AFSUniversity/afsuFrameset.htmhttp://autobox.com/AFSUniversity/afsuFrameset.htm highlighting the difference between Box-Jenkins and Regression.
Which is why it is common practice to use a training, validation and testing set: training for optimization of the loss, validation for telling when to stop optimization. Evaluation of the model (in the very end) on the test set.
It seems a stretch to call a "deterministic enumeration" a sample.
I think so. Just my intuition and experience talking.
@diwgan32: Yes, your comment is correct.
@Glen_b: Thanks. I just had a quick look at the data, & log price is pretty linear in log carat; but price seems to be truncated, mucking up the fit at higher carats.
Biplot? - example on this site, [Visualizing a million, PCA edition](http://stats.stackexchange.com/questions/7860/visualizing-a-million-pca-edition/7862#7862)
I agree. But I wonder if the sign test is really appropriate in this situation. I removed the abstract example and put the real results to clarify things. Could you very mind to check if my solution is appropriate or not?
I meant 20, my brain is random, no idea how I got to 6
+1, nice.  I was going to talk about the case when X was {0, 1}, but I had to run. You beat me to it.
You can do statistical studies with any amount of data, even just one response.  The important question is whether there is any valid way to draw conclusions beyond the sample itself.  When non-response rates are high, one usually cannot say much, if anything, no matter how sophisticated or extensive the analyses of the data may be.  The usual dodge for academic projects is to acknowledge this and then explicitly to pretend the sample is representative and conduct the planned analyses anyway, regardless of how small the dataset may be.
So what conclusion did you reach based on your simulations? Why not show a diagram of the resulting plot rather than this plethora of code? In fact, I would suggest that you delete everything in your question after the line:  `Is Z distributed as well as a Beta-Binomial (with parameters n1+n2, a and b?`
I think that in the normalization TSHalf1 lines, object cbind1 is actually what you named bindTSpair?
Welcome to the site! I think this is not an acceptable answer, since the question actually is: how to avoid conditional (fixed-effects) logistic regression by modified cross-sectional logistic regression, with the purpose of speeding up the estimation. As your reference indicates (at the top of page 3), "we can use either `Stata`’s `clogit` command or the `xtlogit, fe` command to do a fixed effects logit analysis. Both give the same results. (In fact, I believe `xtlogit, fe` actually calls `clogit`.)" The OP already knew `xtlogit, fe` based on the penultimate paragraph.
You can't *prove* it without defining 'good'. (You can prove an asymptotic result, but you can't declare it to be 'good' at a specific sample size without defining your criteria.) You can demonstrate its behaviour by direct example (from which people can see how good 'good' is by their own lights). For typical criteria people tend to use, a continuity correction works well for $\\lambda>10$ as long as you don't go deep into the tail.
@cbeleites "Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." http://stackoverflow.com/help/on-topic Unfortunately this question falls between forums here, in my view.
Consider accepting some answer.
To be honest I am not really understanding your answer. Could you clarify it a bit more and how it relates to my specific question/s?
@user765195 How many total treatments are we looking at?  Did any groups receive multiple treatments? Some example data would be really useful in answering this question.
Also- what is the response variable in the above data (multiple hierarchical counts are available) for a log linear model?
"French pollsters justifying quota sampling for all their work are just being silly, or ignorant of sampling theory, or both." Sounds about right to me.
(+1) Well done. (32-bit only Mac version)
@sino The quality of the algorithm *compared to using all the data* shouldn't depend on the heaviness of the tails.  A fairer way to measure error is this: let $F$ be the empirical cdf.  For an estimate $\\hat{q}$ of the $q$ percentile, what is the difference between $F(\\hat{q})$ and $F(q)$?  If it is on the order of $1/n$ you're doing awfully well.  In other words, just what percentile is the P2 algorithm returning for your data?
I'd like to say, people find Q1 as bad as Q5 for example. Q5 and Q1 are types of things that people can find very bad to very good.
Looks like you're actually quoting IrishStat...could you link the original source of the quote please?
The change is from going from a set to a subset. If I have a table of all 13K customers with an average duration of 1.7 months, and then I just look at the subset of customers who live in Florida, and the average duration is 2.5 months for the 3K who are in Florida, I want to know whether the average duration in the subset is significantly different from the average duration in the whole set. Is going from 1.7 to 2.5 a meaningful change, or just the result of randomness?
Look up "entropy."
Added, thanks Glen.
That works indeed, appreciate it.
Couldn't you just simulate the posterior Gamma distribution from the full conditionals defined by the conjugate priors of the Gamma alpha and beta respectively? Thanks.
what are you trying to accomplish?
Thanks @Henry! I'm grateful to learn another different derivation to the problem.
You should not be too worried about the symmetry of the interval but about its coverage properties. Asymmetry of the Bootstrap distribution is common when the sample size (of the original distribution) is small/medium. Please, have a look at [this paper](http://staff.ustc.edu.cn/~zwp/teach/Stat-Comp/Efron_Bootstrap_CIs.pdf) about the construction of several types of Bootstrap confidence intervals. The ones you are constructing are sort of "quantile-type" CI, but it is better to calculate, say, the $0.025$ and $0.975$ empirical quantiles and construct the corresponding 95% interval.
You have three variables (year, output, capital).  Are you interested in the relationship between output and capital, or is it their individual behaviour over time you care about -- or something else? What *aspects* of the information matter for you? (Is this for some subject or is it a real issue you face?)
IMHO I wouldn't use SPSS for that task. I have been using SPSS and Matlab for clustering (I'm a novice in clustering I have to say) but the difference in time and the flexibility is very noticeable between the 2 packages, being Matlab a better option even if it is also a high-level programming language. As far as I read R should be way faster, but I didn't try it yet. Just my 2 cents.
Are these actually *negative* log posteriors?
I edited the question trying to clarify my problem.
If you are doing this for some concrete data set, you should think about bootstrapping the bias.
Hum, I really appreciate your help for sharing your own experiences. In fact, I've almost finished this project by using the "LibLinear" library, my final system is implemented in Java. R was only used to create some prototypes and evaluate the performance of different kernels. Thanks a lot anyway.
Do you have a precise definition of what you mean by *correlated in the long run*?
Could you provide some justification for your heuristic?
@tomka I have updated my answer.
I didn't mentionned it, but the coefficients were calculated from a meta-analysis model using function `rma` of package `Metafor`. I'm not sure I understand the second part of your comment...
+1, nice to see you again.
Could you please explain how it would be possible for a percentage of *anything* in a group of size $n=6$ to be any value other than $0, 100/6, 200/6, \\ldots, 500/6,$ and $100$?  In particular, how do you arrive at $1.5667\\%$?
Thanks! The study is an exploratory study to get an initial understanding of the nature of those sentences (unlike an experiment, I had no initial hypotheses and treatments). I want to quantify the qualitative findings I had and to inform about potential interactions between the classifications, but I do not attempt to establish any cause-effect relationship. The categories are whether the sentences are {questions,phrases,keywords}, what data set they used when formulating them (3 data sets, nominal scale), and how close they are to the data set (6 levels, ordinal scale).
+1, very good example!
A closely related question is discussed at http://stats.stackexchange.com/questions/15511/estimating-the-intersection-of-two-lines.  In the present case there are simpler solutions because you have specified beforehand the ordinate at which the crossing is to be tested. BTW, there's a good reason the lines shouldn't quite cross: daily fluctuations in temperature cause the true fit to be slightly curved for low heating energies.
It's a Java implementation of Lasso.
Checking individual levels is, in some sense, a "post-hoc" test that is usually done after determining (via some omnibus test) that the variable as a whole is significant. So, I would advise testing the variable and, if significant, consider pairing down to sub-levels (although that may cause interpretability problems). It might be best to go "all or nothing" when doing model selection.
Can you say which CV question you are referring to? Note that an ANOVA is a multiple regression model, just one w/ only categorical covariates. In R, the code / formulation should be identical for performing a nested model test b/t 2 ANOVA's & 2 MR's.
A square symmetric matrix is not necessarily positive definite (the one-by-one matrix with entry $-1$ is a simple counterexample) and positive definite matrices do not have to be symmetric.
Corresponding SO question for reference in case anyone answers over there: http://stackoverflow.com/questions/17945963/wekatesting-a-model-built-using-pca-pre-processing
Normality tests are largely useless for large datasets; the OP certainly does not have that problem. However, tests on datasets of size 10 are not very useful either.
Of course, if you prefer a more commonly used (and more frequently updated) R package `Hmisc`, just run `binconf(5, 12, method = "all")`
@Onturenio, generating bivariate normal variables with the desired (Pearson) correlation and transforming the data will not lead to uniform variables with the desired correlation because the transformation will change the correlation; Pearson correlation is not invariant to monotone increasing transformation, but Spearman's rho is, hence the choice to generate bivariate normal data with the appropriate level of Spearman's rho. The difference between both correlations for the bivariate normal is small; you might use the 'simpler' approach if this (small) systematic bias is acceptable.
This question, suitably generalized, is discussed at http://stats.stackexchange.com/questions/38337/. [Searching our site](http://stats.stackexchange.com/search?q=dummy+difference+interpretation) turns up a few hundred closely related posts: please look some of them over.
(1) You might find [R-squared: useful or dangerous?](http://stats.stackexchange.com/q/13314) informative. (2) Concerning the phrase "significantly influence," you should also read some of the threads found by searching this site for "causality," including [Statistics and causal inference?](http://stats.stackexchange.com/q/2245).
The linked CV question from Huber should address your technical knowledge gap here. The literal interpretation of the Y-axis is a dimensionless quantity, like an infinitesimal grain of probability sand. An interpretation of a smoothed density estimate that's appropriate for applied audiences and scholarly journals is to calculate estimated "probability area" in pre-specified ranges. If the X-axis here were times to recurrence of cancer in months following therapy, you might be interested in risk of 1 year recurrence which you could estimate by using the function `integrate(...)`.
What is the purpose of your document and who are the audiences? I'd start from getting similar articles and look for some examples on how they are done in your own field. I am more familiar with biomedical literature and most of the time, we just use a table. Illustrations are more often seen when the authors try to explain an interaction.
@whuber, do you like it better this way?!
How does your model differ from a multiple regression? I think that's important information. If you just had a set of predictors all correlated and with a set of regression lines to the outcome variable, then you would have a saturated model with no fit statistics.
I think this question is really to soft for the forum but I won't downvote it though
The generic term for this is [Errors-in-variables models](http://en.wikipedia.org/wiki/Errors-in-variables_models). The particular case you're considering looks like [Deming regression](http://en.wikipedia.org/wiki/Deming_regression).
Dan, I have another question. As you know exponential distribution has a tail. my data range for `x-axes` is from `0 to 1`. I don't want this tail in exponential distribution after `1`. Is any other revised other distribution for that?
"Perfect Correlation" implies that there is an *exact* linear relationship between the two random variables.
@user22062: I've edited the answer with a relevant link.
About the correlation issue, I would like to be able to specify that the correlation structure is along time within each trial independently, without having to specify the trials as a random effect of my model. This the same issue as posted here [link](https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q1/003311.html).

It seems that no matter how the groups are defined in the GroupedData object, the groups for the lme object are fully defined by the random effects of the model. I could not find a way to specify a grouping factor for an lme object without it being a random effect.
I just know how to specify two crossed factors with lme(). Could it help ? The method is here http://www2.imm.dtu.dk/~perbb/st113/Module06/R.html (section 6.2.2)
+1 Very good! It looks to me that we had the same idea, you finished your answer while I was still working on mine.
Also asked on MSE: http://math.stackexchange.com/questions/681577/intuition-behind-statistical-completeness
I believe this is the origin of the $.632$ in the bootstrap 632+ rule.
How can I provide my data?
+1 That's a very clear, well-stated summary.  It's a great answer to question #2.
It will work unless the page occupies a minority of the image; then it will fail.
Do we have to assume that you are considering a fixed set of predictors, i.e. you are interested in getting a reliable prediction given $k$ predictors, or are you also interested in some kind of penalization on the $X_j\\quad (j=1\\dots k)$?
@Ymareth Please ignore my previous comment. As I wrote, the median can be defined as the point with the minimal distance to all other points in the set. In other words the median is $y$ such that the sum of $||y-x_i||$ is minimal. This is why the norm won't matter in one dimension.
Using the hypergeometric distribution is _way_ too complicated an approach to this fairly elementary problem.
A big +1 for showing a practical way to figure out asymptotic behavior in general.
Thanks! Good to know I'm not the only one who finds this material difficult. I have a hard time following him in lecture and the notes are a bit skimpy. So we go through and draw our table values from the hypergeometric distribrution. When we go to calculate the importance weights for each cell, what is the equation of that. I'm getting mixed up between distributions which take a table parameter and those which take a normal value
I think this reply may be a little too strong and perhaps loses a chance to focus on a key issue, which is model mis-specification. Take the toy example of regressing $y$ only on $x$ with $x$ ranging in $[0,1]$. In some ways it is *optimal* to design the data collection so half the $x$'s equal $0$ and the other half equal $1$: this minimizes the S.E. of the slope. We don't do this in practice because we need other values of $x$ to check possible non-linearity (model mis-specification). Absent such nonlinearity, the estimated parameters will be *excellent* for *all* $x$.
1. Ideally, yes. This is usually the interpretation of the error term in models. 2. If you are interested in intra-class (or within-individual variation) then duplicates are better than single measurements.
I believe the OP is trying to compute an interval for N in binomial sampling, where p is known. It makes sense to try to do that.
Have you already given a look at the `nlsur` command? (`help nlsur`)
Thank you for the proposal! I will have a look at it!
Sure,  you could use some kind of statistical test to quantify how far from the mean there series gets during 'events.'  In the case of EEM vs SPY it's pretty obvious from the graph... Your re-sampling method sounds like 'blocked-bootstrap' re-sampling, and I think there's R packages to implement that method on time series data.  It's certainly more appropriate than using regular bootstrap re-sampling, but your milage may vary.
@probabilityislogic: No :)  See my answer.
Thanks! I did my dissertation on collinearity - it's been a while (since 1999), but I still remember some stuff :-).
Cost benefit analysis is not a trivial exercise, normally valuing the costs is not that easy, and the valuing of the benefits is extremely hard. Also, a discount rate is used for out-year costs and benefits, although costs tend to be front-loaded. Do you have any economists you can talk to?
I agree with @Peter_Flom. It is usually important to include in the model the option of not buying anything. This makes a third option and it is therefore better to have a multinomial outcome model.
I am not sure that this is the appropriate place for this question.  Other than testing a theory and asking about statistics to do this, I am unsure what this has to do with software development.
@whuber: I think $h$ is just an arbitrary function of $\\theta$, and $h(\\theta)$ is what to be estimated.
Don't you mean DV - dependent variable?