tttjjj/elt_bench huggingface dataset:
Thank you very much for your reply, it means a lot to me. I was testing whether individuals feel culturally similar to those I had within a scenario, and whether that affected their ethical decision-making (i.e., if they felt culturally similar to those in the scenario, did it cause them to think the scenario was more unethical?). The Cronbach's alpha was .566. I had used existing scale items.
@Glen_b that is an excellent point, thank you, and I have edited the answer to provide a more well-known definition. As someone who deals in insurance math, I appreciate that I may be guilty of thinking it its terms, which can be different from more generalized statistical terms.
Correct - the last row should be 1's - fixed. The Eij's are just the known data for Event i variable j (so it's just a real number). I'm using Event in the context of the problem - not in the conventional sense - it is a predictor matrix. The aij are the traditional regression coefficients.
Asking your question again won't really avoid the need to provide answers to the kind of questions I was asking before. The [same things I was asking about in the thread there](http://stats.stackexchange.com/questions/109368/mann-whitney-u-test) will matter. The first and most important issues relate to what your actually trying to find out there. There's also some difficulties to deal with when trying to do analysis that's both multivariate and distribution free.
That's correct - by default trees for boosting are stumps. To see all default settings, click on the templateTree link in the Learners section of that doc page. MinLeaf and MinParent are the two parameters that control the tree size. The doc for MinParent says: For boosting, the default is the number of training observations.
I have merged your too accounts, michelle. Please, register it; this way you will be able to edit your question, get system-wide notification, etc.
I got different naming scheme for all apart from (2).
This question seems to be *only* about how to get this done in SPSS, @Marc Schubert. Do you have a substantive question about the statistical aspects of this? Whether Qs like this are off-topic for CV is in flux right now, but such Qs can always be posted to [SPSS's listserv](http://listserv.uga.edu/archives/spssx-l.html); we also have some other links for internet based SPSS support [here](http://meta.stats.stackexchange.com/questions/793/internet-support-for-statistics-software/807#807). There are 1 or 2 regular CV users who know SPSS sytax, so maybe you'll get an answer in the interim.
Is the "$r$" after the "$\\mathbb{P}$" a reference to $r$ or did you intend to write $\\Pr$ ("probability of") instead? And presumably $b$ and the $x_i$ are constants, right? Finally, your question appears to have nothing to do with stochastic simulation but that is mentioned in the title: have you left out some important details?
Could you explain in what way the p-value is a "*measure of statistical reliability*"? I don't see a sense in which it could be interpreted that way.
I'm estimating them from data. Does that count?
1) Your example data make it seem as if each effect size estimate comes from a different study ($studylab$). Is that correct? 2) You mention a continuous variable, but this is not part of the example data.
possible duplicate of [Should confidence intervals for linear regression coefficients be based on the normal or $t$ distribution?](http://stats.stackexchange.com/questions/29981/should-confidence-intervals-for-linear-regression-coefficients-be-based-on-the-n)
@Glen_b: Under the null hypothesis of uniformly distribution on $S_1$, there is no such point. Under the alternative, yes. But we have to construct our test considering the hypothesis. Generally, without proof nothing is valid.
Given $S_1$ and $S_2$ as variances over $t_1$ and $t_2$ I want to find the variance formula for $T=t_1+t_2$.
Thi is a broken link
@Aaron , Word Shape is something I learned from Stanford NLP course. It basically defines whether the letter is capital or not, and consequent letters in the same case are treated as a single character. You can see some information here: nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/process/… . That's why I used 'Xx'. In other cases, you might define a word as '*ing' or '*ly' as word shape depending on what you are trying to do. It is like feature extraction.
It sounds like you want regression, not correlation.
Thank you very much! It's really helpful!
Better to edit your question to include this extra information than to have it as an "answer" - that can be quite confusing for the reader.  Unless you believe you have answered your own question.
Bonferroni is more conservative when both tests are appropriate. But if your tests aren't independent, you shouldn't use Sidak.
Thanks @gung. I am aware of the technicality that you mention but I was slightly concerned that any discussion of "joint densities" would be a bit of a stretch for "layman's terms"...
"Variance explained" is the between-cluster variance. It is all about ANOVA. Perform one-way ANOVA by factor = cluster variable, for every feature (question). Sum SSerror across the features (this is **W**). Sum SStotal across the features (this is **T**). Variance explained is **(T-W)/(k-1)** where k is the number of clusters.
@BigG, what do you mean by seeing all the stickers with probability?  This might be best as a whole new question (you can link to the one above for background) with more detail on what you are looking for.
@Glen_b : the index of dispersion is coming to 1.15 ( ideally should be 1 for Poisson)
Okay, I've thought about it some more, and I know how to make the argument for products over mixtures when the distributions being added or multiplied together are discrete, but I'm not sure how to apply it to the continuous case Hinton is talking about in the paper you referenced.  Would you still like for me to give it a shot?
I think the idea of "probability of accordance to normality" has flaws. No distribution is perfectly normal, so, what exactly does this probability mean?
if i could give you 10 votes i gave ...
Are you sure this is the question you want answered? What is the practical meaning if you find that one correlation is higher than two others? Is your goal in fact to select variables for a regression equation? Are in fact all these variables in the same dataset? If not, maybe they are not dependent. Anyway, this seems like a difficult technical problem, so maybe it's worth spending some time making sure it's a relevant question in the context of your research goals.
Can you elaborate on how to use the NER and Chunking provided by CRFSuite? It looks like it expects training data of a different format. Where can I find this?
So, if I understand your answer, a factor is an output of factor analysis. A component is simply an output of principal component analysis?
DankMasterDan  --  It depends on what population shift you want to estimate. Normally, it would be the second of those two. If you want an interval for that shift, the second of the two variance terms I gave would be appropriate for the calculation (we're no longer assuming the null, we're estimating the difference). Further, if you want your test and interval to correspond, you'd then shift your test to using that variance as well.
"Stops a random number of projectiles" could mean various things. Could you explain this?  (Asking us to assume a blatant counterfactual such as "sheets ... are not damaged" when they are perforated makes this question particularly hard to understand!  If you are using the sheets and projectiles as a metaphor for something else, you might be better off explaining what you're really studying.)
I don't see that the autocorrelation results add much to your question. What's `dyt`? The data would be more to the point.
You should perhaps scale down your ambitions a bit. You can handle e.g. eigenvalues with Lanczos' algorithm. I've used it with sparse matrices of size NxN with N in the billions, but if you intend to invert such a beast, there is no guarantee the inverse will be sparse, and then you're out of luck, even with a very big computer farm. So you should look carefully at exactly what kind of problems you need to solve.
There are no computational issues.  Choose a time unit that is readily interpretable.
Welcome to the site, @saima. Questions of this sort have been asked many times on this site. We do want people to be able to get the help they need, but also don't want the site to end up being cluttered with many versions of the same question. Please read some of the threads categorized under the `[likert]` tag. You can do so [here](http://stats.stackexchange.com/questions/tagged/likert). If you don't find what you need, come back and ask a more specific question with what you had learned & what you are still concerned about.
Are you familiar with [tag:copula]s?
Welcome to the site, @vrushali. I edited this to make the English smoother; please make sure it still says what you mean. Also, I am not sure if the second set of questions (about adaboost) is the same as the first set of questions; it may make more sense to separate them into different threads.
What about using an extended Cox model with a time interaction for the time dependant variables?
@pedrofigueira: The variance of the e.g. PC1 projection *according to the OP figure* is definitely << 1, so I don't think it was normalized.
@Richard: Sorry guys, my complete story: I need to estimate mean revenue (the population is very heavy tailed). Sample mean is a natural unbiased estimator but has a  wide confidence interval.  I hope to find an estimator with smaller MSE.  I fitted normal mixture to log transformed data. The fit seems good visually. But the point estimator I got (as what Richard suggested before) seems to be different from sample mean. I wonder whether I calculated it correctly and therefore raised the original question. If right, how to reduce the bias?  Thanks for your suggestions!
This list was becoming less usable. It needs to be organized & at least somewhat standardized to be useful. However, there is no reason to think my reworked version is ideal; eg I'm not familiar w/ all the links. People w/ more expertise should feel free to add / change / shuffle or otherwise reorganize.
One way of doing it: A linear regression of clicks on an indicator variable for type of banner, another indicator for position and another for their product.
(oops, subjects when I meant agents).  No, my thinking was that the data is collected in a repeated measures design such that each agent has several loads of each page design.  If you don't care about the agents, only the page design, what you can do is average each design within each agent and do a repeated measures ANOVA.  30 might be too few to eliminate skewness of the kind you're likely observing.  But you can look at the distribution of resultant means and see.
Has anything come of this/will come of this? I'm much interested...
Note that this is *vaguely* similar to rejection sampling. However, in the method you describe, note that your resulting sample size is actually a random variable. Depending on what kind of analyses you have in mind, this may or may not introduce some complications. For example, in many (but not all) GLMs, the sample size is effectively random, but the inference is done conditionally (and this approach can be justified rigorously).
@JDL, ... And if a particular observation has low probability under _both_ null & alternative it seems reasonable _not_ to regard it as extreme.
OK, if it's RM see Masson & Loftus (2003).  Don't do what they do and put error bars on the raw values, only on the effects (as they do).  You might have completely different kinds of non-inferential error bars on the mean values but you need to consider what you're going to say about them.  The effect CIs are very straightforward to interpret in terms of inference, replication, and meta-analytic thinking.  Any error bar around the raw values in an RM design shares none of those traits.
Also maybe if you tell us what the course text was, we can have a better idea as to what might be too complex or inaccessible to you.
@AndyF In this particular case, a classical $\\chi^2$ test (and not a trend test) as the factor `Age` as not been specified as an ordered one.
For multimodal data, tests of location (mean or median) are rather inane.
Yes, it is Durbon-Watson statistic. I am asking in the context of Durbin-Watson statistic.
For a (regular) one-parameter model, there's a reason for Jeffreys prior to provide "good" results, because it achieves an optimal frequentist-matching property.
The "full conditional" that is referred to being sampled in Gibbs sampling is (in your notation) $p(M|D)$; it's a "full conditional" whenever $D$ contains all the variables not in $M$. You can then use Bayes to try to evaluate it in terms of $p(D|M)\\times p(M)$.
...which refers to non-stationary processes, for which actually, "whiteninig" may be simpler, in the case the processes are Integrated (and they need to be also co-integrated).
Thanks for the example and the explanation. I am still a bit clueless about what endogeneity and exogeneity mean in plain English. What exactly do i mean when I say that a variable is endogenous or for that matter exogenous.
@Stas It's not a good idea to make inferences about the person based on the question they ask.  Sure, how a question is posed can indicate whether a technical or non-technical answer would be appropriate.  But it shouldn't *ever* lead us to believe that the proposer is incapable of learning from an answer or not using it effectively.  In this case, on the contrary, "norm" suggests a relatively strong mathematical background and the fact that the proposer is concerned about goodness of fit suggests an awareness of important and subtle issues.
+1 Thank you, that's definitely more elegant (and should be faster, too!) than looping through categories.
An alternative application simply computes the $\\chi^2$ statistic for the two-way table of the data. This one assumes nothing about the marginal distributions of $X$ or $Y$ and merely evaluates independence. I suspect it would not be quite as relevant to you as the first (Poisson) application.
Sorry - did not add comment -- just updated my question...
So you mean to say that the CPI just cannot be considered as a time series which contains a component that depends solely on food prices and a component which is independent of it ? This i am not so sure about. My thinking was that it would fail because there would be a lot of variation in the dependence on the food component. but you have given a whole different line of thought, which i will need some more convincing about.
Well, the way it looks to me, it is not the mixture of two Gaussians. Certainly the upper part is not Gaussian, with all those 100's. As for the lower half, it looks to me like it has only 8 values, so it will be hard to say what distribution fits it best.
Can you relate $\\mathbb{E}(z^2)$ (which you will need to know to find $\\mathbb{E}(Y)$) to the variance and mean of $z$, which is the information you have?
Also, ... it isn't a great idea to call your data `data`. (See here - `require(fortunes); fortune("dog")`)
The presence of the `linetype=2` argument inside the `aes()` brackets - plotting the 99% lines - gives rise to an error "continuous variable cannot be mapped to linetype" with current ggplot2 (0.9.3.1). Amending
`geom_line(aes(x = number.seq, y = number.ll999, linetype = 2), data = dfCI)` to
`geom_line(aes(x = number.seq, y = number.ll999), linetype = 2, data = dfCI)` works for me. Feel free to amend the original answer and lose this.
Actually I think the ending part of the equation ((1+df(t))^alpha3 is not part of the exp() part of the equation. Does that make sense? Would there then be a way of simplifying for regression? Thanks a lot!
@DWin: Thanks, please see my example added to my question.
How are you selecting your slack penalty, and what values do you typically have? Are you getting the reached-max-iterations warning? If you save the exact set of features/labels that you're training on and rerun libsvm independently on the same problem, does it still not converge?
Thanks for your reply. I am sorry for my vague question -I am mainly asking for the Gibbs sampling. In each iteration of Gibbs sampling, we remove one word, compute a new topic for that word according to a posterior conditional probability distribution inferred from the LDA model through Dirichlet integration, (and assuming the topic allocation for all the other words) and update word counts. The reason why it works, is because at every iteration, simply more words are associated to topics that follow the LDA model dsitributions...right?
A common thing to do for the features space is normalize->PCA->train
I am a bit lost. Could you please help me with -- i) What do you want to see or visualize in this graph. ii) I can see that you have a **time** variable, **no. of downloads** variable, and possibly file names (which to me seems like a bad variable to include if it can grow to a huge collection), what other variables are you measuring. Barchart is ok, but you can do better since you have a time-variable with an ordering property. Maybe something like a line-plot.
I might have missed something, but the sum of $N$ **independent** Bernoulli trials is the binomial distribution. Are you saying that your simulation for independent trials has that correlation, or that you want to construct a simulation to have the specified correlation?
@JacquesWainer : Could you detail in an answer how would you do it either way ? How to split w/o looking, and which test using according to whether I looked at the data or not ? Thanks
what sort of transformations are on your mind. Can you give some examples?
That's the part of the question that we would ordinarily send over to [SO]: it's not on topic here. However, many people often volunteer working code in order to make their answers complete and fully helpful, so if you can first describe a method to find the clusters, then the rest of your answer is fine.
Are the symptoms really independent? I would start with checking whether the symptoms do not correlate with each other; I doubt that they don't, but that depends on the symptoms.
@ABC tf-idf by definition refers to the full matrix of weights.  Perhaps you are interested in the idf weights alone, which you would get by `log((number of docs)/(number of docs containing the term))`.  You could also just filter out the infrequent terms.
+1 I would assume that single imputation performs slightly better than mean substitution because you incorporate additional information ($0$ predictors vs $p$ predictors). However, I fully agree that MI is the way to go.
Hi Marcin, welcome to CV. Thanks for taking the effort in making such a clear, well laid out post. However, it's [not usual to 'sign off' with your name on a question](http://stats.stackexchange.com/faq#signatures), since your user information underneath already functions as a signature. As such, the faq (see the link in the previous sentence) requests that you don't 'sign' a post. (I have removed it for you.)
Fixed, thanks for the suggestion, didn't realize you could use mathjax.
The intuition is explained at http://stats.stackexchange.com/questions/14483 and  http://stats.stackexchange.com/questions/71333. The same question (but with various changes of variables $x\\to x^\\prime$) appears in multiple threads found with a [search](http://stats.stackexchange.com/search?q=PDF+jacobian). A subtlety here is that some estimation procedures will be highly sensitive to your use of the max and min of the dataset to normalize it, so it is not clear how you would assess the uncertainty or goodness of fit in your estimate.
Ok, I will asap.
@whuber: I found some time to look over everything, but I don't think I was that far off originally. The CI width is (not inversely) proportional to the pooled SD of the DV in the entire design. In an ordinary $t$-test, this only includes two groups; others are ignored (hence no adjustment for multiple comparisons). Adding more comparisons increases CI widths as $q_\\alpha$ is adjusted, but can either increase or decrease MSE depending on how the additional groups' SDs compare to those in the $t$-test. On that point I wrote too absolutely and may have assumed falsely in this case, but maybe not
Percent of missing values is important, but what's more important on the whole is whether missing values occur at random. Suppose just 1% of values were missing, but they happened to be the highest on some variable. That could mean serious bias. So, on the whole there can be no reliable criterion that so many percent missing is tolerable: it depends on the pattern of missingness.
On Stackoverflow it was put on hold... because of being off topic.
(ctd) ... please see the meta thread [How to ask a good question](http://meta.stats.stackexchange.com/questions/1479/how-to-ask-a-good-question-on-crossvalidated), and consider asking a specific question—one with neither a trivial one-line answer, nor one that requires several pages of text (or code) to respond to.
Statistical terms relevant to the situations you describe are "endogenous stratification" and "truncation".  A search for these in a statistics textbook or on the web may help.
@Bernd-Weiss: yes, I know about the schools/classes :-) I have that problem too. Try to think of every student (student in N) being taught by every teacher (in M) with N>>M, and there are *repeated measures* for students:teachers. Each teacher has a "fixed effect" (i.e. some are "better" than others), and there is also variability of the teacher effect *within students*. And yes, I think that it makes more sense to include a random coefficient for `choicenum` instead of a grouping factor.
@JamesTheAwesomeDude What I was trying to convey to you was the importance of understanding how the checksum algorithm works. In my example, you can see there is a distinct difference between theory and practice because my checksum algorithm is *not* ideal. Also, different checksum algorithms have different purposes. In the most basic sense, an "ideal" checksum would give impossibly small odds of agreeing with reference given any error. In practice, the outcome depends greatly on algorithm used. But in general, yes, a checksum is not supposed to "tolerate" some errors unless designed to do so.
(ctd) ... (Note that a paired design is a repeated measures design with only two measures.) -- the way to deal with order effects in practice is to balance the order of treatment (randomize patients half to one treatment first, half to the other treatment first).
Additionally, in regards to the criteria that i'm using to decide that there could be a sub-factor - well, it is based on a pre-conception I suppose informed by the findings of a similar study. I am hoping to derive groupings (originating from a likert survey of people's reactions to their local environment) that can be used as a response variable in relation to an experimentally manipulated scenario.
@Imorin I had another question http://stackoverflow.com/questions/15656270/markov-chain-for-prediction . If you have any experience working with that please help me out or if you any site where I can get help please share with me. :)
My guess is that most of the residuals are quite small or quite large, with little in between - in turn suggesting that dispersion of residuals is related to some other factors - in the model or not. I suggest looking at plots of residuals versus everything you can think of, and see if you discover where that is happening.
A plot of the histogram might also be useful to inspect. It might look like a skew-t distribution, rather than needing a mixture model.
It would be helpful if you noted what language / program this code is using? Actually, methinks all postings that use code should necessarily do this.
Did you try google first ? for me, it gives http://infinity.cos.edu/faculty/woodbury/stats/tutorial/Data_Disc_Cont.htm
Spot on! I was pretty sure my y was of factor type but i had the factor(y.train) part commented out so it was integer type. Thanks!!
Can you please clarify what you are trying to say in the final two sentences of the post (*For example...*)?