tttjjj/elt_bench huggingface dataset:
I am a statistician.  I can tell you about the probabilities and statistical concepts underlying the likelihood equations and such.  I don't know the specifics of how they program it or anything like that.  Your initial question was what does p'(word) mean and I explained that. Also I can see what score(word) means. That other term in score(prefix,word) doesn't suggest anything to me.
@AndyW Thanks for this and for the clarification as well.
Just found the aodml-function from package "aods3", which implements a beta-binomial regression from a frequentist point of view. The log-likelihood provided by aodml differs only 0.11 from my version, and the AIC only by 0.09.
Thanks for weighing in.  You might be interested in [a recent post](http://stats.stackexchange.com/questions/69898) which appears to offer a real counterexample to your supposition that 30 or more cases will produce a sufficiently Normal sampling distribution of the means. One answer (mine), upon examining simulations from distributions like those cited in that question, concludes that an important issue to address is *power*.
**Always** is a pretty strong statement.
Why pay $70 for the book?  To support the authors and to have a physical copy.  That's why I did it, anyway!
+1 I hope you will forgive my adding an appropriate link. If the OP is actually interested in providing evidence of absence, they might explore equivalence tests, e.g. [TOST](http://stats.stackexchange.com/tags/tost/info).
More important than either is to know why you are looking at these data in the first place. Without a context and a statement of objectives, we are left with an ill-posed, open-ended math problem and it is impossible to know how to give good statistical advice. So please edit this question to reply to these requests for clarification.
I edited to perhaps clarify more.  You were referring to, "the unique contribution of the amount of broadband connections over and above the amount of broadband connections/country."  If you don't understand that sentence that you won't be able to interpret the resulting regression because that's whan an effect of broadband connections would mean with both predictors in the model. If the whole idea makes no sense then the regression makes no sense.  I think it's a rather convoluted and confusing way to go about the whole thing and probably doesn't make sense.  Which is why you're confused.
@Brendon Independent, the article happens to have information on that as well.
@armundle see addendum please.
Why not transform your data?  Take logs of both axes, maybe try something more extreme...
@DWin, in terms of code, you are correct; I probably should not have marked that up as code. I was using $\\ln$ as a base $e$ logarithm in contrast to $\\log$ as a base 10 logarithm. In `R`, the mathematical function $\\ln$ is `log()`. So it should either have been $\\ln(2.9)=1.06$ or `log(2.9)` which gives `[1] 1.064711`
@Alexis is correct in emphasizing that assumptions such as linearity must be justified, whether by domain theory or by model checking. However, I think the outright deletion of outliers *without carefully considering why such values occured* is a very common error in statistical analysis.
$\\frac{\\sum (x_i^2)}{n}$ converges in probability by WLLN?
Yes, the two stage approach (Gelman method of two seperate models) assumes two populations, those at zero and those > 0. Regarding the hypothesis tests, can you frame them in terms of the predicted values for varying levels of the inputs and construct empirical confidence intervals related to the simulations for each? For hypothesis tests for the coefficient != 0, you need to test this seperately for both models.
Ah, I see. Then the problem is same as a problem of minimization of $D(\\hat{p}\\|p)$ over all $p$ satisfying the marginal constraints, because $\\sum_{ij} c_{ij} \\log p_{ij}=N^2 \\sum_x \\hat{p}(x) \\log p(x)=N^2(-D(\\hat{p}\\|p)+\\sum_x \\hat{p}(x)\\log \\hat{p}(x))$.
Google "polson and scott normal variance mean mixtures" - this will give you some approximate bayes using MAP via em algorithm.
I can't error-check it mentally and I don't have the time to compile it. I suggest you test it with synthetic data and check the reference code I just provided.
Paired-preference models have been discussed on these related threads: [Testing the importance of an item among a finite set of items](http://bit.ly/qGJQtn), [Multinomial choice with binary observations](http://bit.ly/pMt0Lr). The problem here, as pointed out by @whuber in his comment, is the sample size.
Let me start from here, density is the derivative of distribution and CDF is the integral of the distribution is that correct ?
Rabiner's review paper on HHMs is an amazing source for starting to work with HMMs. All the important algorithms are described (forward algo, backward algo, and EM algo) Take a look (http://www.cs.cornell.edu/Courses/cs4758/2012sp/materials/hmm_paper_rabiner.pdf)
How is PCA involved in the above description?
Ah, I was close ;). Thanks a lot @Alecos. I really appreciate all your help! Enjoy your evening.
Caltech assignments: http://work.caltech.edu/homeworks.html
@mpikat: I'm not aware of this theorem, could You provide some more details about those conditions? I was thinking about a linear transformation $f:R^{n}\\to R$, but it is not obvious for me how to estimate parameters of such transformation.
@lcl23, what is $B$? If it is a coefficient from logistic regression, why are you interested in $\\exp(B)$?
Thanks, mpiktas. I figured there must've been a mistake somewhere in my pdf.
Don't worry, just get on with the work.
might want to look at how researchers have investigated this question. A classic is Gilovich, T., Vallone, R. & Tversky, A., The hot hand in basketball: On the misperception of random sequences. Cognitive Psychology 17, 295-314 (1985).
But how can it be the same, when BIC and AIC use different penalizations? This sentence that you posted makes no sense to me!
It would help a lot if you could post code (either here or in a gist or something) for a reproducible example so we can see what you're doing.
Thank you. It's a pleasure to be part of the community!
Could you describe what you are doing precisely (which dialog box, etc.)? My – possibly outdated – understanding was that multilevel models were not available through the GUI in SPSS. For step-by-step guides to the SPSS sides of things, see my answer to http://stats.stackexchange.com/questions/24662/how-to-organize-the-data-for-this-repeated-measure-within-subject-setup
Be careful what you do with the results then.  For example, if it were the case that after B became 1, it hung around at 1 for 30 seconds, then the probability you get from a naive method will very biased if interpreted as the "probability that any randomly chosen location with A=1 is currently in B=1 at any given time".  However, if your intention is only to apply this result to exactly the same data collection algorithm in the future, then it should be OK.
@zkurtz Can all the forecasting methods incorporate causal variables ? The reason, I used cuasal varaibles as one of the criteria is that not all forecasting method that I know of can incorporate causal variables.
Including an interaction without the main effects is almost never right. And the wide range will also affect the interaction.
How many grid cells do you have? And how large are the cells?
Calculate a regression model with both variables PLUS the [interaction of the two variables](http://www.theanalysisfactor.com/compare-regression-coefficients/) (humeral length $\\times$ antero-posterior diameter). The interaction tests the assumption of parallelism of the slopes of the two variables. If the interaction term is significant, the slopes are different.
Thank you for your answer.  Sorry, I kind of accidentally blended two questions there-- first, I wanted to check whether a higher interval proficiency test score directly corresponded to a higher ordinal Likert score for global competence; so, judging by your answer, it looks like I should run the one-tailed Spearman's correlation test.  Another goal is to check the inter-rater agreement between the six raters for each of the six ordinal variables scored for each of the 24 score sheets passed to each rater (6 ordinal variables per sheet, 24 sheets per rater, 6 raters in this study 6 X 24 X 6).
Presumably 1 IV means one independent variable and 3 DVs means three dependent variables. Abbreviations and acronyms should be explained, please: see http://meta.stats.stackexchange.com/questions/1479/how-to-ask-a-good-question-on-crossvalidated/
This has been answered many times.  The short answer is no.  But instead of asking if a certain parameter makes sense to examine, state what you would like to estimate and go and estimate that.  I find this is most easily envisioned through differences in predicted values.
Are you using $\\sigma = E (X-EX)^2$? I am more used to seeing this written as $\\sigma^2$.
This doesn't sound like a sensible approach to subset selection to me.
@MichaelBishop I don't think we need to explicitly refer to SPSS *in the title* as we already have a tag pointing to that software.
Wow, thanks this is indeed a good starting point! I already played around with scatterplots yesterday, but the faceted one by task is very good! Thanks
I couldn't agree more with you. Unfortunately my company deals mostly(if not only) in week-based time series modeling. I believe that using months would be much easier for us though.
+1, I'm also trying to figure out the difference between the Confidence interval and the Credible interval, and I found your ball-drawing example more readable than other examples in other related posts. Just one quick question, I repeat the experiment m times, collected m different samples, so now I've computed m different Confidence intervals (each with 95% confidence level), now what is Confidence interval? Does it mean 95% of m CIs should cover the true value?
`?MASS::theta.ml` (the same page covers two other estimation methods) - The first example is for estimating it for a GLM; the second example (while still using a GLM) does it for the case where all the data are assumed to follow the same distribution.
Ehrm. Sure. Read my answer from the start until "... is obvious."
That is because that answer does not exist in general; it is necessarily a judgement call. In situations like these I like to run simulations. That is a great way to develop an intuition for the problem, and you can use that to get an answer that is tailor made for your data and problem.
Good answer, but it's not that 'fairness' is hard to pin down - 'equiprobable & independent outcomes' seems to do the job - but that the multifariousness of *unfairness* forces us to focus on specific directions of deviation. Fair coins are all alike; every unfair coin is unfair in its own way.
I included a link to the Lipsey/Wilson book "Practical meta-analysis" (see References). I am a bit worried that PLINK's and my results differ. Do you know what meta-analysis methodology they use? You also should take into account that I have absolutely no idea about "genome-wide association studies".
Extrapolation of a fitted curve is something one should only do very carefully. Your extrapolation is only as good as the model that you used to fit the curve in the first place. If you have a mechanistic model in mind (i.e. this is some natural world phenomenon and you can model it from sound theoretical principles) then you can do it. Otherwise here be dragons. If you want some more help, please describe in words what the data means.
I would just like to remind you of the cautionary notes included in [my answer to the other question](http://stats.stackexchange.com/posts/6038/revisions), because they apply here, too. How *I* would tackle your problem would depend strongly on *why* you are fitting a distribution to these values and on the size of the dataset they are estimated from. Regardless, neither *ad hoc* methods (such as yours) nor "standard" methods such as a least-squares fit are truly appropriate due to the correlations among these statistics and the different levels of error that occur in measuring them.
Thanks Sir!,I am agree with your answer.Basically reliability is authenticity of a construct /Test/Questionnaire while validity is the degree of trust worthiness ,soundness and accuracy of data that gives information out of it( test/questionnaire).
I can graphically see on the time course that high-frequency, low-ampitude components which are irrelevant to my measurement disappear after I downsample. In any case, *assuming* that what I do makes sense, is there any "better" order in which to derive and downsample? and if so where's the difference?
Check out the third picture: http://stats.stackexchange.com/questions/3947/help-me-understand-support-vector-machines/3954#3954
I presume the green table is probably a subset of the full parameter estimates (re: second paragraph)
This should possibly become a community wiki.
Reducing complexity is always helpful when there is not much data.  But isn't the easiest way to reduce complexity to reduce the numberof variables in the model.  The OP was not suggesting cutting out any of the 5 variables.  he only suggested adding data and in a very artificial way that does not help.
Check the following simulated example: `set.seed(1);
data = rtpareto(100, 1,10,2);

ll = function(par){
if(par[1]>0&par[3]>0&par[2]>par[1]) return( -sum(log(dtpareto(data,par[1],par[2],par[3]))) )
else return(Inf)
};

optim(c(1,10,2),ll);` . (Requires VGAM)
Also the dependent variables are percentage of gonad reproductive stage, so they are inter-related: a percentage of gonad in atresia will affect the percentage of gonad in mature stage. How to account for that? Thanks in advance, Tania
Thanks @Glen this is helpful!
1) The noise part is an AR(1) with a non-zero mean
This is clever, but are you sure it will work as described? It seems to me the appropriate approach would be to remove a trend of (1.6+1.4)/2 = 1.5 and then control the ridge parameter until the *absolute value* of the slope is less than or equal to 0.1.
@Jacob I get that. Sorry. In the definition of the covariance function for noisy Gaussian regression, there are two variance parameters. Your $k_f(x,y)=\\sigma^2_f G(x,y)$, where $G$ is the standard Gaussian with variance 1. The variance of an individual observation is the sum of those variance parameters, and their relative magnitude controls how noisy the regressions are. When $\\sigma_e$ is large, the correlation between neighbouring sample points is diluted. However, I have looked at the code, and the rest of the gpml toolbox, and I don't quite see why sn2 is defined the way it is.
Thanks this is very helpful. Could you clarify this a little more for me? "...is to give zero prior support to sigmoid functions that don't have detectable saturation on both ends of the data domain". How would you give zero support to non-asymptotic data?
It simply makes no sense whatever to apply a chi-square test to these measurements. The technical reason for getting NAs is because for the months March, July and December the expecteds you're generating are all zero. Their contribution to chi-square is a sum of terms like $(0 - 0)^2/0$... but they're meaningless *anyway*, so it's actually a good thing the analysis doesn't work; better no answer than a meaningless one that appears meaningful - that's *dangerous*. Whoever has asked you to do such a thing apparently has no idea what they're doing.
Frankly Adrian, that's bad advice. What if the simple method doesn't achieve significance? *Then* change the test? That's an unfair way to test a hypothesis. Neither of us know how much work the OP cares to do (that's less clear from the OP than the existence of repeated measures, which seems pretty plain to me), so the existence of two alternatives is good, but let's not discourage the more difficult approach just because it's complex. IMO, that instinct is one of the primary problems with statistics as practiced by people who are less than serious about it.
Thanks a lot! Got the materials and will work on it. :)
Why do you want average annual temperature to look normal? For that matter, what are you using correlation to do - simply to answer the question "are they related?"... ? If that's the case, the ordinary correlation may not be a good choice for how related they are (it would seem that something that measures monotonic association might be more relevant). But presumably this isn't random sampling. What is it you hope to show?
(+1) As a first approach. Note that youtube's policiy for counting views is not well understood given that they have not made their algorithm public. They only say: "A view is counted whenever someone watches a video on YouTube. We do not get more specific than this to avoid attempts at artificially inflating view counts" [(see)](http://www.youtube.com/t/faq).
Not sure what you mean - is it that there are some cases that don't occur in the data? Those *do* cause some concern in the asymptotics of the tests, just like in a chi-square situation. You kind of have to grit your teeth and don't get too literal in interpreting the results.
My guess is it's really something pretty ad hoc, like finding those variables having a correlation with the outcome variable greater than some threshold such as 0.5. It seems like trying to duplicate something Excel does in R is kind of an empty cause: it's so easy to do something much better.
thanks. I considered your point on shared ratings, and I think it is a good idea.
Like I mentioned in the post, what I did is basicly a normalized Minkowski Distance.
The un-normalized formula does take into considertion the shared ratings.
What I'm trying to do now is to add that consideration to the normalized metric.
Do you have any ideas on how to accomplish this?
@uvts_cvs, I gather you have found that this response answers your question. You may want to *accept* it by clicking on the check mark on the upper left of the answer to acknowledge this.
One approach for dealing with multicollinearity issues is to use a ridge regression or lasso available in the glmnet R package. Considering that you're looking at multiple dependent variables that are correlated with each other suggests using structural equation modeling (sem package in R). Have you identified which variables ought to be influencing other variables according to principles of plant biology? Drawing up a "cause-effect" diagram would be helpful when applying SEM.
I know about Tukey's HSD. I think I was unclear in my question. While i'm doing ANCOVA, i'm not trying to generate CIs from post-hoc pairwise comparisons. I'm just generating them by bootstrapping the samples in each level of the factor (also tried using the t-distribution). The CIs are for the mean of each individual group (factor level). I agree with Tal's comment that this does not allow differences between groups to be assessed. It's easy to do a Bonferroni correction on these CIs, is it possible to do the Hochberg (or other) method? (p.adjust requires a vector of p values I don't have).
+1. Adding the constant $2\\log(r)$ to the log-likelihood will not change the location of its maximum, but converts it into a function of $\\sigma/r$ and $n$, whence the value of $\\sigma/r$ that maximizes it is some function $n\\to k(n)$. Equivalently, $\\hat\\sigma=k(n)r$ as you claim. In other words, the relevant quantity to work with is the ratio of the standard deviation to the (observed) range, or equally well its reciprocal--which is closely related to the [Studentized range](http://en.wikipedia.org/wiki/Studentized_range).
I guess that's why this question is tagged [tag:Homework] :-).
I would say no. The deviations are simply on a different scale. They are equally anomalous within the scale of the data.
It seems to me that you didn't use clustering. You used multidimensional scaling (mds). So, was the whole question about mds, i.e. about plotting the distances between the 25 shapes in euclidean space?
What's "lag dependent variable"?
Can you then comment intuitively how would I interpret the relationship between a respondent's factor score and his choice on the good, which is my variable of interest with 4 possibilities? I am trying to learn the proper interpretation of respondent's factor scores as these random numbers mean little to me by itself so I need to find a way of relating them to my dependent variable. My goal is to eventually use them in a multinomial logit, but I can't do that without a proper understanding of the scores themselves.
In essence, the function generates data from a (frequentist) model of your data, incorporating uncertainty about the actual parameters. It could be part of a Bayesian MCMC routine, but could also be used in [simulation-based power analysis](http://stats.stackexchange.com/questions/35940/simulation-of-logistic-regression-power-analysis-designed-experiments/35994#35994) (nb, power analyses based on prior data that [do not take the uncertainty into account are often optimistic](http://stats.stackexchange.com/questions/18028/desired-effect-size-vs-expected-effect-size/18172#18172)).
@Glen_b, thanks for the tip!
A nice answer by @Glen_b. Please see the FAQ (frequently asked questions) for information about this site. You can expect well-thought-out answers about specific questions you may ask about statistics, but you shouldn't expect other users to conduct your analyses for you or to teach you how to use your software (at least, not for free). For those things you may need to hire a consultant--but this is not really a place to seek out that kind of help.
One quick thought is to bring the 68-95-99.7 rule (http://en.wikipedia.org/wiki/68-95-99.7_rule) into the discussion somehow.
By the way [here](https://class.coursera.org/ml-003/lecture/64) is a Coursera video that explains learning curves extremely well.
LASSO stands for "least absolute shrinkage and selection operator" so it refers to the L1 penalty term, so I'd say it was a LASSO method whatever the loss.  I'll edit my answer slightly.
You probably should consider providing some data and a minimal example.
I have one more question regarding per-modelling or data prediction? Do i need any any data-cleaning process before using ELKI, R or WEKA tool or i can directly use my :CSV file directly on software. Actually i am student and and i have to use twitter and flickr realtime data but my teacher give me  instruction that i have to see the data and evaluate the data statistically and graphically to see the no of clusters on dataset and on the behalf of result i will use algorithm. I am not from this field that's why i am fuzzy to solve the problem?
With a huge training set or a continuous input stream of data, you can use iterative algorithms like Stochastic Gradient Descent and grab the input in small batches as you move on. Is that what you were asking?
Others should be able to advise on #2, but the answer to #1 seems intuitive. The procedure doesn't know that your response is non-negative and the amplitude of seasonal variation is increasing over time. More broadly, expecting a time series model to pick up on whatever external factors cause production to shut down at irregular times seems a real challenge here.
The Central Limit Theorem is about the journey and the Strong Law of Large Numbers is about the destination.
There is no way possible that it can be "clear" there is an interaction effect. I don't doubt that you interpret what you see as an interaction, but it is logically not possible to determine whether there is an interaction. You need more data. A lot more.
Extreme-value theory suggests that even a huge number of observations of the min and max of $100$ variables will reveal little about their distribution. Absent strong, quantitative assumptions, there's no hope of making extensive inferences about the full multivariate distribution based on such a dataset. As an example, let $X_1\\sim$ Uniform$(-100,-99)$, $X_{100}\\sim$ Uniform$(99,100)$, and let the $X_2,\\ldots,X_{99}$ have *any* multivariate distribution $F$ supported on $(-99,99)^{98}$. These data give information *only* about the trivariate distribution of $(X_1,X_2+\\cdots+X_{99},X_{100})$.