tttjjj/elt_bench huggingface dataset:
1. There is only 1 value per line. Some of the values are in the thousands/millions. The value represents the "importance" of the item. 2. The rating system is really for curb appeal for a consumer facing website.
Thanks, I should be more clear about my goal. By clustering I want to do some kind of Co-reference resolution in text. That means as you said my similarity function doesn't guaranteed to be triangular. Specifying number of clusters isn't my preference but the gold clustering that I'm looking for should results fine-grained clusters. How can I use appropriate indexing for DBSCAN or OPTICS methods? I tried using *scikit-learn* implementation of DBSCAN, but It computes whole similarity matrix that I don't want.
Thanks thats quick! So for each subject I indeed have 6 preference ratings, and 6 accuracy ratings. Accuracy is the mean hitrate over 16 identification trials (16 for each type of fruit). A hit is when they select the right fruit, miss is when they select the wrong type of fruit. Hope that this made it more clear.
I think you should consider your data per condition, for example consider studying patient weights across a range of ages, where a weight of 15 stone could be reasonable for a twenty year old male, but not for a ten year old girl.
If I understand this correctly you have a two-way table for the 45 cases studied by both people. See http://en.wikipedia.org/wiki/Inter-rater_reliability for an approach common in several fields.
as far as I know, a kernel based NB is just a Naive Bayes which uses kernels to approximate the pdf of continuous variables, meanwhile "default" NB assumes a normal distribution (or make the data discrete beforehand). In this spirit, is the paper linked in this question/answer sufficient ? [Use of kernel density estimate in Naive Bayes Classifier?](http://stats.stackexchange.com/questions/4298/use-of-kernel-density-estimate-in-naive-bayes-classifier)
@Tawani - they are not all missing the point. What you say needs to be defined using generic terms. You cannot go with a single example. Without general definitions, if 400 is 30 is it still an outlier? And if it is 14? And 9? Where do you stop? You need stddev's, ranges, quartiles, to do that.
You should describe what the inputs and outputs of your data are, even in basic terms. For instance, when you say "logit", do you mean binomial or multinomial logistic regression?=
I realise that there are countless variants of this algorithm. What I was interested in was which ones do people routinely use.
the algebra is: f(x, y, z) = x + (y - x) / (z + 1) where: x - is the modeled rate (typical value between 0 and 1) y - is the contribution of every new record to the learning about the rate (typical value 0 or 1) z - is some kind of convergence/learning speed parameter (typical value any integer between 30 and 900)
As an aside: another idea would be to do multiple analyses on random subsamples of the big dataset. That way you can also cross-validate.
The Azzalini [skew normal](http://en.wikipedia.org/wiki/Skew_normal_distribution) is a popular distribution proposed in 1985. It has support on the whole real line.
Hi schenectady. Thanks for your answer. The comparison was planned (by others - without statistical input I gather). It's human work and the drop out rate was rather large for various reasons. Basically I'm trying to help out as I'm the 'guy who knows R'. I'll have a look at the linear model lit but I expect it's beyond saving. Hey ho.
This is really useful stuff but I think I'll need to bit of time to get my head around it and actually use it. Could you have a look at my other question? http://stats.stackexchange.com/questions/56501/histograms-and-cdfs
I'm suprised no one has brought this up: Mann-Whitney U test is non-parametric test. You would usually utilize this test (instead of the t-test for example) when you are concerned about the assumption of normality holding for your data. Look at a histogram of your data. Does it look normal? If it is clearly skewed you should go with Mann-Whitney. If its pretty close to normal you should be fine with t-test.
Are you adding interaction terms when you add your dummies? What software are you using, & what output are you looking at? Often there is an ANOVA table, & a parameter estimates table; eg in R, there is `anova(model)` & `summary(model)`. These two can provide different information & so be potentially confusing.
I would also add that I bought "An Introduction to State Space Time Series Analysis (Practical Econometrics)", which was a very gentle introduction to the topic, and it too, used SsfPack/Ox. Makes it much less useful, in one sense, though it does have the benefit of once you implement it in dlm (in R) yourself, you can check your answer against an independent implementation.
I don't mean to answer anyhow because you are telling little about your data (why 'categorical-data' tag?) and you goals. But I may notice to you that in _factor analysis_ (do not confuse with PCA) 43% of overall variability explained is not bad at all.
Thanks for the answer. This along with the link has been most helpful !
Parallel lines assumption seems to be crucial for ordinal logit model. If the assumption is substantially violated, I would 1) try another link function; if it doesn't help, 2) try maybe merging some adjacent categories of the DV. Else, 3) consider the DV as nominal and use multinomial logit model.
The baseline doesn't have to be zero. It could be e.g. 20°C for an area chart showing heating/cooling degree days.  But your argument seems right - there's no sense in emphasizing an area under a curve when the product of the variables on the two axes doesn't have a meaningful interpretation. On the other hand, does it do any positive harm?
Excellent point (+1).  An example of 'raw signal': radar/sonar systems often have 4 detectors slightly offset, each continuously returning a voltage v. time signal.  The phase difference between these 4 signals is then used to determine the angle from which the source came.  This kind of processing is **real-time**, i.e. has to be done almost at machine speed, and requires analog to digital sampling, one of the cornerstones of signal processing theory (Nyquist theorem, etc.)  By contrast, business time-series are almost always part of a data understanding phase, and not hard-real time.
please, read the highest graded answer (i.e. mbq's) here: http://stats.stackexchange.com/questions/1001/spearmans-correlation-coefficient-to-compare-distributions
@Glen_b yes I know how to do chi squared questions, but in this particular type of problem they haven't given me the expected value or probability and only the observed value of the survey is given. My school friend told me a vague answer that u have to use binomial O.o to find the expected values, As for your first comment my class consists of only 2 students and the teacher doesn't like me nor do I like him and it's now holidays and I can't ask him.
On the other hand, if "input" and the associated "output" are interpreted as *subject*, and there's lots of different possible inputs and outputs, then algorithm becomes like group or treatment, right? While CS has historically been interested in boundaries and limits of performance, in some applications those things are of more theoretical interest, since most inputs and outputs are located inside some continuum.
Do you mean causality? Generally ordinary regression only measures the association. To obtain causality, you need to do some causal inference. You can search "causal inference" to find the criteria for causality and the usual methods.
I cannot and will not give you a "statistical blessing". But, based on the (necessarily) very limited information I have about your problem, I would lean to consistency.
No variable that has standard deviation 0 could possibly be correlated with another (non-constant) variable. Correlation is a measure of how large/small values in one variable correspond to large/small values in another variable - if one of the variables is equal to a constant with probability 1 (a consequence of having standard deviation 0), then it can't possibly give information about whether the other variable is small or large. I don't know what the convention is but it seems like the correlation should be defined as 0 in that case.
"So I just wasted a week of my life on this." That's called learning and research. Being a master student you should embrace that because there is more like this coming. There usually are no shortcuts in research because often no-one knows the way!
(+1) I think this answer is starting to get closer to the crux of the matter. There are actually *two* questions from the OP, posed as one (at least, as I see it). The first is (**1**) Can a person without a PhD do meaningful work in industry in data-mining and/or machine learning? The answer is undoubtedly affirmative. The second is (**2**) Can one *obtain* such a position with limited experience in the area and without a PhD (and how hard is it to accomplish this)? Even though the second is probably easier to measure quantitatively, it seems to have more grey area.
@Peter It sure looks right-censored to me.  Which side the censoring occurs on is of little import, because by negating the dependent variable one switches between right- and left-censoring.
There seems to be no predcit() method for logit.mixed in zelig..
I think these figures are in the right ballpark generally, but are off by a couple percent. It's difficult to tell since a description of the calculation you've done hasn't been provided in the post itself.
Also, can you explain the repeated measures nature of the design. Are the two conditions repeated measures or are you just saying that you measured the DV at multiple time points and that is what makes it repeated measures? You may also want to comment on how many time points you have and how that relates to your conditions.
The question is very unclear.
The basic idea is that an antiderivative that can be written in terms of a finite number of "elementary" functions has to have a pretty particular form. When you take $e^{-x^2}$ as the function for which you desire to find an antiderivative, you then show that the particular form needed cannot actually be found.
+1 The reason you run into trouble for certain values of the parameters is that Gamma has poles where $a-k$ and $a+b-k$ are negative integers and the hypergeometric functions have corresponding poles and zeros to cancel them, leading to finite values. You can sneak up to the correct value by taking limits (which *Mathematica* will do); they will involve derivatives of the confluent hypergeometric function with respect to its parameters (the first two arguments).
Just a quick check: do any of the independent variables have missing values? If so, that alone can cause the r-squared statistics to be incomparable.
possible duplicate of [Machine Learning: Linear classifier and possibility to separate](http://stats.stackexchange.com/questions/26471/machine-learning-linear-classifier-and-possibility-to-separate)
Could you maybe include 1) the plot and/or 2) a dataset so that we can reproduce your example?
ah... if that is the case my answer is probably a delete...
I should just point out that the Bayesian intepretation of probability is not necessarily subjective, so that is not really a strength of the frequentist approach.  In the case of the objective Bayesian approach to the "probability of a coin landing heads up problem", using an uninformative prior involves no subjectivity at all.  The real strength of the frequentist approach lies in problems such as quality control, where is is natural to talk of repeated trials and long run frequencies.  It only has difficulties when you pose questions about particular events.
But @ndoogan, the original question was about proportions, not successes/failures.  And the above code does work, as I took it from the `cbpp` help page.
Does the bootstrap-estimated median have less bias/variance than the sample median? If yes why? can you give me a proof?
Out of curiosity, why two answers, rather than editing one answer?
What is the point of a VAR in this case? Why not just use a univariate ARX model that incorporates the contemporaneous and lagged values of the other variables as well?
I've often found a Bayesian approach using JAGS, Stan, etc. to be very helpful in performing these kinds of more unusual hypothesis tests, particularly in the context of hierarchical models. For example, if you could integrate it into one model, you could get a posterior density on the difference between the two coefficients.
Pie charts on scatterplots actually work pretty well, at least if they are simple (no more than 3-4 sections in each pie), because *en masse* they are visually interpreted as graduations of color across the plot, enabling almost instant apprehension of interactions (even nonlinear ones) between proportions (3-4 variables) and location (another 2 variables). There are few (if any) other effective techniques to discern such interactions for proportions.
See also http://cran.r-project.org/web/views/Distributions.html
I actually meant the population filter applied to abs($x_1$), as $x_1$ can take negative values. But I guess your answer is still relevant.
It is still a bad title. You should use something like "What is the minimum sample size to have good power to detect a significant difference?"
I am determining the probability of entire observation sequence (one symbol sequence at a time). I was wondering If I have to combine the probabilities of multiple sequences. I'm using [jahmm](http://jahmm.googlecode.com/svn/javadoc/0.6.2/be/ac/ulg/montefiore/run/jahmm/ForwardBackwardCalculator.html) library.
Since you seem very competent in SVM, let me ask you to clarify my doubt: once we found the best separating hyperplane, what do we use it for? We can define SVM as a method that, firstly, chooses the best hyperplane to correctly classify data points, and, secondly, it uses this hyperplane to sever new data points in the two classes. Right? (I've some doubts on the second part)
Thank you very much, Simon, the first paragraph explains what I needed to know... +1
3. [Handbook, co-edited by Hedges](http://www.amazon.com/Handbook-Research-Synthesis-Meta-Analysis/dp/0871541637/ref=sr_1_2?s=books&ie=UTF8&qid=1348050925&sr=1-2&keywords=meta+analysis+Hedges)
+1 for a properly posted homework question.
Actually, k-means works decently--provided you can meet the constraints. I tested this out in a prototype about ten years ago using simulated annealing to find the solution. But I'm not really writing to defend k-means; rather, to point out that contrary to your first sentence, this *is* a clustering problem and would benefit from being analyzed as such.
Don't forget to center & scale the data!
This does not appear to be a chi-squared statistic. The correct statistic uses the *model* value to estimate a *variance* in the denominator. Wouldn't it be more constructive to explain what you are trying to accomplish instead of asking us to fix a procedure that looks so completely broken to begin with?
This is a really nice reference. Thank you.
Cronbach alphas are healthy as per the manual. They are all in the range of 0.7-0.9. And are stronger with in factor even in samples. Could this be a sampling problem, after all these results are not shown in norm group. But again how can the same error be replicated in three different samples?
Thanks. However, patients are nested within physicians in both A. and B. So that doesn't seem to resolve it.
@Shea Parkes - You're right. The defaukt number of trees is 100 which was not enough to generate a good model. With 2000 trees the partial dependence plots of gbm and random forest are almost identical.
An admirable sentiment.  I like to quote Krane, John (2005), “A Physicist on Wall Street,” when he calls such things "voodoo".  It isn't science.
... to expand on BabakP's comment, if, for example, you're using lm to estimate the model, the predict function is predict.lm (that's what you'd type in w/o parentheses to see the code.)  If you're using "glm", then predict.glm ... and so forth.
(**-10**) for an inappropriate language!
"*such as when the treatment effect is expected to decrease between each sample*" -- I'd have thought the relevant thing to do would be to model that decreasing effect (with the reduction in noise/uncertainty) rather than base it off individual, noisier points. You *could* do it point by point, but you wouldn't do that the way you suggest because (if I understood it right), it ignores the noise in the treatment sample.
Looking at the denominator, are you specifically asking about the addition of new data that change the variance of a particular variable?
GEE is not a mixed effects model, they estimate different parameters. The repeated measures ANOVA and GEE with compound symmetry are very similar as you noted, but GEE estimates marginal effects while mixed models estimate conditional ones. The interpretation is thus different, for a GEE the $\\beta_M$ is an: "associated difference in the outcome among individuals having a unit difference in predictor" whereas mixed models say $\\beta_C$ is an "associated difference in the outcome for an individual having a unit difference in the predictor."
@B_Miner 1) The other way around - to get the proportion right, you need to set the intercept correctly - so adjust the intercept until you get the proportion which you are expecting. 2)  The logic of ytest is that we need to get a dichotomous 0 or 1 outcome.  So so we compare each sample from the uniform distribution to the probability (prob) to get our dichotomous outcome.  The 'runis' doesn't have to be drawn from the random uniform distribution - a binomial or other distribution may make more sense for your data.  Hope this helps (sorry for the delay in reply).
@TayShin, you can *accept* his answer by clicking the check mark below the vote total. Later, when your reputation is >15, you can return to upvote.
@Tormod Very different in what sense? Running the `ex5.5` example in Mplus and R/ltm, I got approx. the same results. Some variations depending on software being used are to be expected, though. Usually, what I've found is related to the choice of the parametrization (logit/probit), whether it's $\\theta_p-\\beta_i$ ($i=1,\\dots,k$ items, $p=1,\\dots,n$ persons) or $\\beta_i-\\theta_p$ that is modeled, and the estimation method (marginal vs. conditional likelihood), and for 2+ parameter models whether thresholds are centered or not.
Normally PCA of a data matrix gives out eigenvectors of columns of the data. If you transpose the data, PCA will give eigenvectors of the former rows. No matter p<n or p>n, only min(p,n) first components are nonzero, only their eigenvectors have numeric sense, all the rest eigenvectors can be safely set to zero. Indeed, some programs do the zeroing or just cut those eigenvectors off.
No. ARMA is not a model for nonstationary time series. As part of ARMA culture, you first stationarize the data then apply ARMA.
Simulation of what? "There is no single Monte Carlo method; instead, the term describes a large and widely used *class* of approaches." http://en.wikipedia.org/wiki/Monte_Carlo_method#Overview
+1. The variances of the parameters & their covariance can be found by examining the variance-covariance matrix of $\\boldsymbol\\beta$, which most statistical software can provide. Eg, in R, it's [?vcov](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/vcov.html); & in SAS, `covb` is added [as an option to the model statement in PROC REG](http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_reg_sect033.htm).
Can you tell us about the complexity of the problem (how many rows/cols), the nature of the network (nodes, functions, topology) and the metadata on the training (epochs, learning parameters, which gradient descent, stopping criteria) for your problem?  Without knowing that it is nearly impossible to understand the problem.
Oh right.  Fixed now under "see also" links, thanks for pointing that out!
+1 Very nice intuitive explanation
...in which case you are done since you are left with $E(e^{u_{T+1}}|\\Omega_T) = E(e^{u_{t}}) = e^{\\sigma^2/2}$ - the equalities follow from your assumptions on the error term. Then you have the theoretical formula for the expected value of Y, which you will make computuable by using the estimated $\\beta$s, and estimated $\\sigma$. Alternatively for the $e^{\\sigma^2/2}$ term you can use $\\frac 1T\\sum_{i=1}^{T}e^{\\hat u_t}$
Welcome to the site, @Guest555. You needn't sign off on your posts (in fact, we'd prefer you don't). Notice that your username, your avatar, & a link to your user profile page are automatically added to your posts. Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info like this.
Also, I'd be concerned about that **R^2** value if I were you because greater than 0.94 is *extremely* high. Definitely go back and make sure that you haven't misspecified the model somewhere. Lastly, consider jubo's second bullet point (it may help a lot since right now your approach seems pretty **ad hoc**)...
I thought as much, Victor.  A better guess could be had by (say) splining the data (and a much better one via statistical modeling of the process).  When a price is moving straight through (from min to max or max to min), a uniform distribution may be reasonable; but when it peaks, the distribution will likely depart from uniformity.
But beta would be used to model a proportion and not a count variable on a finite set of integers.
You don't need $\\lambda^{[1]}_j$ for $\\lambda^{[2]}_j$ at all, you need it for $\\mu^{[2]}_j$. On the changing dimension -- that's rather important information. Then you need to use something more sophisticated than a plain Gibbs; exactly what depends on the model and how you want to try to implement it. For example, you might have indicator variables that say whether particular $\\mu$'s and $\\lambda$'s are present and include those in your model and sampler (you will have to integrate some variables out - especially the variables that the indicator can set to 0), ... ctd
@Glen_b, if it is still your will, please let's promote this case :)
I gave you a more detailed answer, maybe this is more specific! Youre welcome.
@Macro, Even in the orthogonal case, you see why the (naive) $p$-values would be off, correct? You also see why $|\\hat\\beta|$ of the "selected" model would tend to be (i.e., stochastically) larger than in the unselected case, correct? Say you had just two orthogonal variables, both $\\beta_i = 0$, and your model selection was to choose the one with the lower $p$-value to keep (same as choosing the one of largest magnitude).
Have you tried the default (and generic) way, i.e. calculate MPEG4 descriptors and cluster?
Doesn't it require a normal distribution? (or close to)
Thanks so much for the wonderful examples and how-to for the `rms` package @COOLSerdash and Nick.
I think that's what I answered.... but perhaps not.

Your answer is also right.
That is not how I understand the term crossed random effects. Though given your current edit, I think you are not looking for random effects of any kind at all.
+1, this is a really great answer!
Section 12.2 Testing for Serial Correlation of Wooldridge's Introductory Econometrics has a good discussion and has an implicit answer to the question: http://www.eco.uc3m.es/~jgonzalo/teaching/EconometriaII/SerialCorrelation.pdf this is close to what I could tick as the accepted answer. Anyone willing to summarize?
@Glen_b, You're right. I have corrected it.
Maybe I don't understand the notation, but I don't get how $P(n\\leq n^* \\mid b)$ is even defined, as it would require averaging over $s$ but there is no probability distribution defined for $s$.
A nonparametric option is the [Kolmogorov-Smirnov test](http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) but there are probably some ad hoc tools for the case of Gaussian mixtures.