tttjjj/elt_bench huggingface dataset:
+1 Thank you for the pointers. I was not aware of Cytoscape. I will try that. And yes, it looks like k-means with cosine similarity is referred to as Spherical k-means. I applied this k-means after applying SVD and reducing the number of dimensions. The way I reduced the number of dimensions was to use the variance rule (pick the singular values that contribute to 95% of the variance in the original data).
Thanks for your response, Peter!
So, I assume then that first I would have to log the variables (all of the predictors?) and, after that, I would center only the independent variables required to calculate the interaction terms. 
One more question: Would you recommend centering or standardizing the variables?
Again, thanks a lot!!
If you have a copy of Hosmer & Lemeshow, *Applied Logistic Regression,* then check out the chapter on "Assessing the fit of the model": the conditional variance of the Binomial shows up *everywhere* and is explicitly accounted for in almost all the GoF tests.
Cross Validation has nothing to do with this problem.
Nice guess. Haven't thought about that. However in my quick 50-fold test there was the "same" 85% TP-rate.
I imagine many here will be unfamiliar with the details of this. It may help to explain a little more about what Rayleigh fading is, and how doing it by finite mixture exponential works.
... but first make sure you get the normalizing constants correct.  And then consider whether there might be another change of variables that *really* simplifies the integral.
@B_Miner Not overly simple at all, that's exactly what you're doing :)
@whuber No I don't suppose that, and I don't understand where did you see that in what I wrote. I said "the map is not the earth", which means "what is in the model does not exist in reality", so how can you infer from that the exact opposite? The "imperfection" does not refer to the axioms of probability, but to what tools these axioms lead us to, and to how effectively these tools can be put to use in order to model, study and understand the real world. And it is obvious that I believe that probability is an effective tool.
Agree in most part: in small sample sizes any approach will have issues, and there are trade-offs no matter which one you choose. Rarely you can avoid hyperparameter optimization, unless you assume that some kind of default parameters will work for the problem. Sample size per se, is not enough to decide best method, it depends on the problem that is how data behave, but also their dimensionality. Otherwise there would be tables like from 1 to 10 samples use method x, for 11 to 101 go for method y...
Given that financial time series are so difficult to model, I don't think it's a good idea...
Huge thanks Procrastinator, code works like a charm! I want to fit a Pearson V as I'm looking at landslide areas which are thought to follow and inverse gamma pdf. I'm now investigating whether other features of landslides do the same. So, although other distributions may be appropriate, it's good to start out comparing like for like . I didn't immediately share my data as this is unpublished work and I couldn't even do a fit to synthetic data. Also, I was originally just looking for a solution to fixing pearsonFitML (which is probably a simple syntax thing!) although I got a better answer!:)
a significant p-value for the Jarque-Bera test indicates that you can **reject** the null hypothesis of normality ...
Why do you think something that converts percentile ranks into something that's nominally a z-score will leave you with an interval scale?
@AlecosPapadopoulos I think it would help the answer to put the thing we _can_ say in a formula like you did with the false formulations. I guess it is something like \\lim P(\\frac{1}{n} \\sum X_i) = 1 ?
Typically classificators based on Bayes rule take such a "constraint" into account, they call it prior probabilities for the classes. Of course a classification will usually violate this request to some degree; you could then re-classify some objects to a second closest class, to satisfy the request fully.
Perhaps some of you good folks could also contribute an answer to a question about Bayesian and frequentist interpretations that is asked over at [philosophy.stackexchange.com](http://philosophy.stackexchange.com/questions/8275/interpret-bayesian-probability-as-frequentist-probability).
Since this problem is routine bookwork, as one might do in a homework exercise, you should add the `self-study` tag and read its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info).
R works utilizing your RAM only naitively. So 3 Gigs allow for roughly (3 * 1024^3  / 8) 4 *10^8 total elements on any given time in double precision format. That's roughly equal to a 20k-by-20k covariance matrix. The number of processors is irrelevant on how big your dataset can be.
$Z$ is one-dimensional. The median of Z is a real number $m$ such that $P(Z\\leq m)=F_Z(m)=1/2$. Hence, if you want an analytic expression for $m$, the first step is to compute $F_Z$, the distribution function of $Z$, and then work with it. In general, there will be no closed form for $m$.
@DWin I have replaced the phrase "out of" by "divided by" in order to make the original intent clearer. A 'count divided by' something *can* be a rate. The original 'out of' was intended to convey the same meaning while emphasizing the connection between rates, and proportions-in-the-binomial (but was, I admit, unclear to the point of being potentially misleading).
Can you have a control group (ward)?
Why did you log transform them in the first place? OLS regression makes no assumptions about the distribution of the independent variables.
@Stat Not sure. Did you calculate a different $\\hat{\\beta}$ for every point in the 2-dim table?
I took it a bit further for you.
Firstly, this looks to be a homework question, so please tag it as such. Secondly, this is a finance question that would be more appropriate on the Quant Finance site, not here.
You may find this question of interest: [what-are-examples-where-a-naive-bootstrap-fails](http://stats.stackexchange.com/questions/9664/).
As in http://gates.comm.virginia.edu/rrn2n/teaching/gantt.htm ?
Do you know the parameters or are you estimating them?
(+1) for the good review! However, it seems that you have answered your own question within your own question...
A very interesting textbook, it appears... Please provide the referrence
Physical theories usually make specific quantitative predictions. These serve as null hypotheses. Your case of "no evidence" is often actually a case of *rejecting* the null. The present question asks about what can be concluded when one *fails* to reject the null.
JohnRos, I am particularly thankful to pointing out the clarify the expectation step. Now, I see the connection between convex combinations and taking expectations (since probabilities sum up to one), and now the connections to convex optimisation is much more clear. Thank you.
Plus adjusted R2 is a horrible thing.
This may be off-topic but sounds like a fun project! From the number of wheel rotation, you can approximate how many calories the animal burnt. You can then couple this data with a feed container with a weight sensor to calculate the calorie in take. If calorie intake > calories spent, Biscuit is probably putting on weight.
Is there a way to move the question without reposting it?
Add another column with p_t-1 values and use `Covariance` between p_t and p_t-1? You will find this tool in Excel on Data (tab) --> Data Analysis --> Covariance.
@guy Good, thanks for letting me know. Recall that a $\\epsilon$−final version of the papers from IMS is always posted on arXiv: http://arxiv.org/abs/1202.51600.
This is off-topic for our site, since it's a question about a statistics package. I will migrate this over to Cross Validated (the Stack Exchange site for statistics).
Can you please make your question a bit more understandable? It seems as you want to use Cross-Validation to make classification itself, but this is nonsense.
Kyle - thanks, I have heard this as well and have been wondering if this might be part of the problem. So, what we are doing is taking two groups of 25 subjects (high risk and low risk for a disorder) and looking at various ROIs & their connections in the brain. Our matrices are 247x247x50. I posted a comment above to answer some other questions that may provide you with a little more information. 

I believe the p value is associated with the accuracy statistic.
Welcome, @OverKAnalytics,  to the board! Great blog articles BTW.
You can put a link to your image, we will convert this as an inline image for you.
Many people would seek insight by *exploring* these data rather than testing them.  Have you drawn a scatterplot?
Thanks all, the for loop approach worked for me. I changed my code to calculate the stochastic notes in c individually. I also changed c in my R code to be a numeric vector. for(i in 1:n){
  c[i] ~ dnorm(mu[i],tau) 
  mu[i] <- inprod(x[1:3],y[i,])
 }
I asked a similar question about how to deal with missing data when you also need to perform some kind of variable selection. Here it is, in case it is helpful. http://stats.stackexchange.com/questions/46719/multiple-imputation-and-model-fitting
@Danu I think the OP actually means there's a functional relationship (with noise) between two variables which looks something like a Gaussian curve, not that there's a sample from a normal distribution.
Similar question:http://stats.stackexchange.com/questions/26722/calculate-transition-matrix-markov-in-r
Have you read [this paper](http://dx.doi.org/doi:10.1016/j.jml.2007.12.005)?
Hi onestop.  I imagined this was the case, but couldn't quite see that from the wikipage (http://en.wikipedia.org/wiki/Tukey's_range_test), any suggestions on references for how this is performed?
Stepwise selection is not recommended for any model, including ordinary linear regression. If this doesn't make sense to you / you want to understand why, you may want to read my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856).
It's *binomial* estimation.  (There is no marking or recapturing at all. which leads to hypergeometric estimation.)
Also, I was wondering about whether I also need to account for the topics being nested in visible/invisible? Or is this the wrong way of looking at things?
+ I remove highest 5% distances after sorting distances in training sample. Is that true?. If yes now I should remove highest 5% distances in out sample (5%-after sorting) ?
I'm afraid I don't follow any of this because I don't see where the twos and eights come from.  According to the information set forth in your question, the new program is 2/0.5 = four times faster.  If that's wrong, then please edit your question. To say "25% faster" is inconsistent with the other information you have offered: a program that is 25% faster does 5 calculations in the time that it used to do 4. I hope you see how confusing the use of percents can be in this context.
Thanks. Yes what I'm trying to do is compare the RW results with other methods. I tried your specification, but I obtain the same results I had with the loop. Maybe this specification is too much naive?
I'm running into a similar issue -- can you please describe in more details (preferably a reproducible example) on how to set a minimum size of members per cluster? Although hierarchical clustering will let you prune small clusters, this has to be done manually, and I haven't come up with a succinct, straightforward way to do this automatically.
As a side note, next time don't put spaces in between your columns when you upload a dataset--it actually kind of messes with the data-loading process...
There are a number of problems like this that are solved in detail on our site. Among them are http://stats.stackexchange.com/questions/81626, http://stats.stackexchange.com/questions/87385, and http://stats.stackexchange.com/questions/48396.
@Fred, this is yet another example of mixing up the "fixed effects" as understood by say biometricians (the linear predictor) and "fixed effects" as understood by economsits (the within estimator in a panel data model). Things are much simpler here, and don't really call for the complicated estimators. Charlie, +1.
Are these two matched samples? What is the scale of measurement (e.g., ng/ml for Vitamin D level)?
Maybe Wikipedia will be sufficient here: http://en.wikipedia.org/wiki/F-test#Regression_problems
Even it re-opened, an answer to this question would be of the same form as the answers in the linked thread. Try reading through it & see what you can learn. Then come back here & edit / update this Q to state what you've learned & what you still need to understand. Then this Q would no longer be a duplicate, & the answers you get won't just be copies of the answers there.
Thanks Nick, I still haven't gotten my head around how S works enough to have figured out the `mean.default` trick and was stuck on the `mean` generic error message.
@Macro, yes, exactly.
Why would you want to include a user ID in a predictive model? As for other categorical variables with cardinality larger than you wish when you use dummy variable coding as you describe, I first run them through a decision tree as the only predictor - in order to collapse the levels. Can also re-bin by grouping "rare" levels etc.
SPSS always rescales it, step1= 1 step_last= 25. You can't change these units to original colligation values taken from agglomeration history, as far as I know. Of course, you could do it manually in any graphical editor.
Sorry, I don't use R. Furthermore, this is not the place to ask programming questions.
@MYaseen208 Your new matrix is not positive definite: it has negative eigenvalues and isn't a possible covariance matrix. I don't know what ellipse to draw in that case.
I'd guess it depends on the size of your model, i.e. the number of latent factors. Can you comment on that?
"Large matrices of such user-item preferences tend to be sparse" Density is of prime importance I'm not sure what long-tail stuff is but nonetheless, I am doubtful it would lead to an unstructured full dense matrix.
@MarkDollar I've merged your two questions as the second one was merely an illustration for the present question. It's better to keep everything in the same thread.
I disagree with you, Macro, mainly because I'm confident that nuts vary in weight enough for this variation to matter.  If, say, your nuts average 1 gm with an SD of 0.1 gm and the weight is 100 gm, then you likely have between 97 and 103 nuts. This is a small range, to be sure, but it's going to affect the chi-squared test. Better solutions are readily available.  (Use MLE.)
Actually, I found what I looking for by using the Cumulative Frequencies column of the frequency tables. That'll do for the problem at hand.
A famous example of this concerns mapping cancer mortality rates by administrative region within a country (such as counties within the US): both the highest and lowest rates tend to occur within the least-populous units. See, *inter alia*, Gelman and Price, [All maps of parameter estimates are misleading](http://www.stat.columbia.edu/~gelman/research/published/allmaps.pdf). Note that this is not a *sample* size effect; it's a *population* size effect.
The first option is not strange at all. It just might be difficult. $F$-score is clearly a function of $a$ and $b$ (for different $a$ and $b$, you get different features, and different $F$-scores). The challenge is to write this function in closed form. After that, it is just a search problem (say, using steepest-ascent.) But that's the same story in the second option. They're both search problems. It's just that in the first option, the search is a bit more guided (you go in the direction in which the gradient is maximum, which will hopefully get you to the global maximum.)
Don't both of those cost a lot of money? R is free, & not too difficult to learn.
To avoid problems of practical significance and statistical significance, you could use an effect size metric. I would recommend something like the common language effect size (CLES). This is rank based and can be interpreted as the probability of a random individual in group A being larger than a random individual in group B.
I guess that's extremely hard to interpret. How are the other p-values? Maybe you should just say something like "if there is a difference it is probably here and here". Or you couldn't perform a different test (instead of a mixed model ANOVA) and compare the results. I don't have a suggestion of a test, but I'm pretty sure other people will have good ideas.
Thanks for the reply. Still having trouble because 'bootstrapping', 'robustness' and 'sampling with replacement' are new terms to me.
This regression approach estimates $\\mathrm{E}(\\theta_Y | \\theta_T)$, which suggests a Bayesian aproach: Specify a prior for $\\theta_Y$, say $p(\\theta_Y)$, and together with $\\theta_Y$'s likelihood, say $L(\\theta_Y | \\theta_T) = p(\\theta_T | \\theta_Y)$, this yields $\\theta_Y$'s posterior, say $p(\\theta_Y | \\theta_T)$.  Using notation for the random variables $\\Theta_Y$ and $\\hat\\theta_T$, we could write the posterior more explicitly as $p(\\Theta_Y = \\theta_Y | \\hat\\theta_T = \\theta_T)$.  I'm not sure whether this is valid or how to implement it.
Thank you very much for your answer. If I understood correctly, the only reason to use Bernoulli model is "A change of representation". Does it mean, the change of estimation, when the estimation was changed during classification and there is a need to recalculate the classification?
Very pretty! I appreciate the separation into an idea that is distribution-independent, followed by specialization to a particular distribution.  Although the other two parts of the question might not yield to the same strategy, your example nevertheless should inspire us to look for similarly clever ways to simplify them, too.
I feel this question is more _algorithmic_ than statistical. Of course it is possible to compute number of concordats and discordants from a frequency table; software computing Gamma or Kendall correlation do just this. (Once I wrote a code doing this in SPSS software.)
This question is unclear: if each observation has "values $1/2$ and $1/2$ respectively," then there's no issue concerning the mean.  So what are you saying?  What does "prob" mean?  What kinds of "observations" do you have?  What is the purpose of computing a mean?
I apologize. I haven't understood the point. Using 1-5 columns and then using `2-`6. Why not 6-10 ? [In this video](http://www.youtube.com/watch?v=53MVU07j6dY) , once they have taken three digits of 1st,2nd,and 3rd column, they move on the next three column starting at #4 column for generating the next random number.
Welcome to [crossvalidated.com](http://crossvalidated.com). [Markdown](http://stats.stackexchange.com/editing-help) is enabled and this is a $\\LaTeX$-aware site.
Wikipedia makes an interesting and (I think) useful distinction: a *macron* is intended to lengthen a vowel. As such, it has an inherent lexical and phonological meaning. An *overline* simply refers to the typographic mark. Clearly the former is not applicable to statistical usage, whereas the latter is.  Similarly, the *vinculum* has the specific purpose of uniting symbols. (I learned algebra long enough ago that the textbook included a vinculum along with parentheses and brackets as a way of grouping terms in algebraic expressions.) That, too, conflicts with statistical usage.
Ain't there a bunch of books on epidemiological statistics? E.g. Rao, Miller, Rao (Handbook of Statistics Vol.27, North-Holland 2008), Chowell, Hayman et al. (Mathematical and statistical estimation approaches in epidemiology, Springer, 2009)?
@DilipSarwate I cannot explain it because it obviously does not hold in general. Thanks. I am editing the answer.
@Glen_b I have added the pattern of restrictions in the question.
Yes, it fails. The sample mean of iid Cauchy's is again Cauchy with the same spread. Thus if you multiply the sample mean by root $n$ as in the CLT, you get a distribution with infinite spread instead of a nice Gauss curve.
Not sure I understand the part about correlation on pre/post measurements: linear or rank-based correlation provides no information on the level of agreement between two raters.
Apparently `it` is a pair of subscripts, but what does `× 2010 i` mean?
If you want to get a predictive model that gives some sense of when orders have stopped, you would need data where you identify when orders have stopped in the past, and the prior activity of the people whose ordering stopped, compared to those who didn't stop (and to their own activity earlier still, if possible); that kind of information would allow you to build some kind of a model for what you want. The Poisson is only going to be even a bit useful if you model the *change in order rate* - but even that doesn't tell you anyone has actually stopped, only that the order rate might be low.
Sure. But you need to restrict the support of $\\theta$ in the likelihood, or as @BabakP mentioned, a prior could help here to regularize the estimation so that it is well-posed.
Isn't it $R^2$ rather than $r^2$ ?
$\\beta$ is a constant that I don't need for this problem. I removed it from the model. Yes, $e^{\\eta}$ is a lognormal ditribution. @Jacob Mick, I don't actually get your suggestion.
I agree. I have many statistics books that are R based, and MASS4 is probably the closest to what you are looking for, but in places "terse" becomes un-readably terse and requires most statistical background knowledge than I have.  That said, I have the book nearly 10 years and I keep going back to it and learning new stuff.  I wouldn't let its' age put you off. Oh, and I'm now doing a stats phd :-)
What is your loss function? For instance, how does overly optimistic overshooting compare with pessimistic forecasts? In the restaurant industry, for instance, it might be more OK to have ingredients rot than to run out of them. Is being off by a factor of 4 worse than two times the cost of being off by a factor of 2?
This is the right idea, but may need some improvements. (1) you should consider smoothing the second derivative to avoid the potential for a large number of spurious results. (2) There is no need to divide by the differences of the x-values because they are constant and therefore will not change the signs. (3) It's faster (although still somewhat imperfect) to identify locations of sign changes with an expression like `which(second[-1]*second[-length(second)]<=0`.
(So, broadly, are the values of the parameters set at the stage of Reality, or the Model? And we use Data to approximate to Reality, or the Mode?)
Look at Anomaly detection technique.
"Randomly pick a number" is not precise enough to allow this to be answered. If the computer picks a number from a discrete uniform distribution, then the odds of picking a 50 are indeed, 1 in 100. But if it picks a number from any continuous distribution, the odds of picking a 50 exactly are infinitesimal. If it picks from some other discrete distribution, the odds depend on the distribution. I suspect you meant the first thing. As for your other question: Do you mean *exactly* one person picking a 50? Or *at least* one person picking a 50?
Hi Pitone, have you tried running it and doing cross validation? You could compare the misclassification rate to other classifiers such as random forest. (I'm not sure the 100 image rule is a rule, the number you need will be dependent on how messy the classification boundaries are)
Do not know, never used this package. But I think in your case `lm` function is sufficient.
There's no distributional requirement on the explanatory variables (IV's, $x$'s) in logistic regression or any other GLM, nor in ordinary regression. Even when you are in a situation where something is assumed to be Gaussian, conditioning your analysis on a formal hypothesis test isn't necessarily helpful. It doesn't really answer a useful question (the useful question is about impact on your analysis -- and the hypothesis test really doesn't answer that).
Start with a problem that has data.  It will contextualize the inevitable math.