tttjjj/elt_bench huggingface dataset:
General comment on your code: You should build your corpus once and then pass your processed corpus into your classification functions. Don't call `create_corpus` anew each time you want to run a classification. You're unnecessarily repeating work.
Please DO NOT cross-post.
For supervised methods, cross-validation is generally a robust method, although you can get over-fitting if you optimise the cross-validation error too much (see e.g. http://jmlr.csail.mit.edu/papers/v11/cawley10a.html) especially if there are many degrees of freedom (kernel parameters).  This can probably be made to work with kpca (especially fully probabilistic versions), but I haven't used KPCA in anger.
An old question, but something about the answers puzzles me: in many circular statistics cases, it seems that you have to consider that 0 and 360 are the same and that 5 and 355 are actually closer together than 5 and 20. Does it not matter in this case?
Did you use **the same scaling** for both training and testing set (e.g. did you use the same coefficients for training and testing)? If not, that is probably what's going on here. Also, any reason you're using the sigmoid kernel? It's rarely used, usually RBF is a better choice. Did you tune hyperparameters ($\\nu$ and kernel parameters for one-class SVM)?
[Here's a decent blog post on change-point analysis](http://www.r-bloggers.com/changepoint-analysis-of-time-series/). Also, a related post here on CV: [Change point analysis](http://stats.stackexchange.com/q/59755/32036). Plenty of references listed at [Changepoint.info](http://www.changepoint.info/introduction.html), including a book ([Chen & Gupta, 2012](http://www.springer.com/birkhauser/applied+probability+and+statistics/book/978-0-8176-4800-8)).
Can you clarify the model you fitted please? Its very hard to tell what things we're comparing here.
<<I've written a program to analyze the time it takes to query <<a data structure and generated a graph which shows on the Y <<axis, the time a query took and on the X axis the number of <<items in the structure.

It does not match what you have plotted, which looks more like a measure over different time of the day. Can you clarify that point ?
Check out the book by DeBoeck & Wilson http://www.amazon.com/gp/aw/d/0387402756?pc_redir=1397798695&robot_redir=1 and this review on fitting models in R http://epub.wu.ac.at/4010/1/resrepIRThandbook.pdf Also, as you said, the work by Rabe-Hesketh http://www.gllamm.org/ and Bartholomew & Knott http://www.amazon.com/gp/aw/d/0470711108?pc_redir=1397026499&robot_redir=1
Are these the same carcasses at both stages? If so are you able to identify a particular stage 1 carcass at stage 2 (in order to associate their values together)?
Continuation here: http://stats.stackexchange.com/questions/53045/monette-foxs-conditional-hypotheses/
Here's a link to a certification discussion over on [Not Programming Related](http://programmers.stackexchange.com/questions/44/are-certifications-worth-it) -- also a Stack Exchange website.
Not as it stands, you have no degrees of freedom in the current setup. How about trying a penalized regression.
@Glen_b The title edited to be more explicit, thanks.
If you have errors in your design variables, ordinary least squares is not very good. The usual LS estimators are actually biased in this setting. A good way out is to use a [errors-in-variables](http://en.wikipedia.org/wiki/Errors-in-variables_models) model, in which case you also should get the answer to your question for free.
Thanks @Vitomir. I like the intuitive arguments that you point out, and your approach is similar to how I approached the problem, but I was looking for more of a formal proof. E.g. a condition which proves that after performing transformation $g$ onto $\\beta_{1}$ in the first regression, then $t_{b} = \\beta_{1} / SE_{b} = g(\\beta_{1}) / SE_{g(b)} = t_{g(b)}$. Best!
@WesleyTansey I realized one could use whuber's idea of considering the transition probabilities to construct a predictive model for the purposes of my answer. I edited my answer to contain some initial ideas, but I have not tried implementing this nor am I currently planning to.
A rw model is y(t)=y(t-1)+a(t) . If you divide by y(t-1) you get [y(t)/y(t-1)]=a(t)/y(t-1) or [y(t)/y(t-1)]=e(t) . Your model is very ad hoc and probably can be made more generic as you are assuming a very specific structure. Time Series model identification attempts to empirically identify the structure rather than assuming the structure. Such a data set as you describe might even pose a challenge to "an old guy" like me BUT that is what I enjoy where data suggests the need for improved methodology.
@jerad good point, let me re-write the Q in a more easy-to-follow way...
@Gurzo Ok, I mean you can impose to keep a certain number of cases from each class.
sorry, I mis-spoke. My sampling dates were less rigorous last fall; in the spring all of the dates were 10 days apart, excluding the first-second observations with dt = 13, but there were no changes between these observations. In the fall however, Oct-Nov was quite rainy; both leaf senescence and the sampling intervals were dependent on weather. (I know that leaf senescence is dependent on weather from biology, this information is not in the data).
@B_Miner Yes, right. I mainly add the "appropriately defined" qualification just to point out that, in cases of more complicated datasets and correspondingly more complicated models, there are many ways to define an ICC depending on what pairing of observations you are interested in.
Many years ago, the East Wing of the National Gallery of Art in Washington DC had a beautiful (and unintentional) example of a normal distribution showing on an exterior wall where two exterior walls met at a 45-degree angle instead of the usual 90-degree angle. People presumably had _touched_ the edge to see if it felt sharp, and the smudges from their fingers left a stain on the wall which showed as a bell curve (rotated 90 degrees clockwise) at about chest height. On a more recent visit, I found that the exterior walls had been cleaned and the smudges had disappeared.
I would *really* like to hear the rationale behind the downvote on this question! The only possibility I can imagine at present is that there was some concern about what the statistical nature of this question might be. However, it seems clear to me that there is a statistical component given that the sales data can be viewed as random counts from some underlying distribution. I suppose an edit that draws this point out a little more clearly might help. But, my comments here are quite speculative. (+1)
What do you mean exactly by "where every out of diagonal entry of R is the same number in (0,1)"? Can you give an example, such as for p=3?
Presuming this is EEG data (or fMRI, or similar,) the number of responses indicated per person is about a relatively poor signal-to-noise ratio in finding e.g. an event related potential signal amongst the neurological (and other electrical) activity one can pick up on the scalp. The answers below are good, but I would strongly recommend that you consult with a statistician with expertise in both EEG and statistics -- because of the signal detection issues for the brain signals, you can't necessarily compensate for fewer trials per person by having a large sample size of participants.
Do you want to know the mathematics or simply a code solution in R or some such?
I don't follow the nature of your data. It sounds to me like you have *counts* that were adjusted (with 'distance' somehow). Is there a max that is meaningful a-priori? Eg, if I flip a coin 10 times, I may get 5 heads, but there is a very clear sense in which 10 was the max # of heads possible (discrete proportion). Or if an organ is taken from a cadaver & weighed, & the tumor is excised & weighed, the max # of grams possible would be the mass of the organ (continuous proportion). OTOH, if this is how many squirrels there are in a forest, it's not clear what the max # of possible squirrels is.
@AlexWilliams definitely. You can't just run nested F tests like crazy, you need a target. Lucidnonsense, yes, that's why this statistic has an F distribution.
I suggest you edit the content of the question to more accurately reflect the context. I tried reading the previous post and am still confused as to what you're getting at.
@nico Although you are correct, your reply is far too restrictive. The need for assumptions depends on how ANOVA will be used and for many purposes it suffices that the *sampling distributions* of the means depart not too badly from Normality. Note that further on in the Wikipedia article you reference it flatly states "there are no necessary assumptions for ANOVA in its full generality."
I think you should look into bootstrap. There are packages about this in `R`, e.g. `boot` and `bootstrap`,
Are you saying that, say, F1-M2 was observed in condition "A" but not in condition "B" and that perhaps F1-M3 was observed in condition "B"?  That is, each female should have been observed in both conditions but perhaps was paired with different sets of males in those two conditions?
"*I could calculate the median but I want to exclude the outliers.*" --- well, the median ignores the outliers (except that it notices which side of the data they're on). If you actually want to *calculate* the average (i.e. the mean) arrival time (which as worded is a sample quantity), you *can't* ignore the outliers. If you define it as the only the average of under normal behavior, how can we discern whether a large value is from 'normal' behavior or from abnormal behavior, unless you supply that information as well?
*Both* variables are ordered. An ordinary chi-square will potentially be throwing away a lot of power, though it depends on the exact hypotheses you're most interested in.
Please make the reference more precise than "a book".
Not quite sure what you're asking. If the aim is to find estimators based on data $Y$ for the constant but unknown quantity $(\\beta_0,\\beta_1)$, then we can certainly *denote* such estimators as $(\\hat{\\beta}_0,\\hat{\\beta}_1)$. More important would be what the expression is for this estimator!
Embarrassing - thanks @Dan! Fixed now.
Then delete this one.
By the way, when calculating the variance of non-linear function (i.e. e^x), how do we do that with formulas and properties of variance?
I haven't actually checked the typical shape of my distribution, I think it varies week-by-week depending on where people make changes. It might in fact be highly skewed.
Nick, thanks for your answer. 

I'm clarifying my position. I'm not asking anyone to do my homework. I'm only asking for leads, to show me the way like when you are in a foreign city and you don't know where that famous place is. However, using that analogy and agreeing with you, I have to do the walk to that place , no one is going to do it for me.
"not all males" rather than "no males"?
thanks for that info.  I see the dbinom() method works for 2d6:

> 11/36
[1] 0.3055556
> dbinom(1,2,1/6) + dbinom(2,2,1/6)
[1] 0.3055556
> pbinom(0,2,1/6,lower.tail=FALSE)
[1] 0.3055556
I see.  Well, if one were generally comparing the 251 and 322 with some GLM with a negative binomial model, you could use the same sort of model to compare the ratios.
Keeping in the text mining theme, you might also be able to, depending on your data, reduce your feature set (i.e. vocabulary) by filtering with Inverse Document Frequency, for example.
@amoeba Yes, that is certainly true. You would be using your reps to validate the model. The model being so validated would be the usual sort of model, however. You would still need some sort of criterion for determining fitness. You could do exploratory FA on half the data and a confirmatory on the other half, I suppose.
You are confusing study design with data analysis.  There are reasons to over- or under-sample groups to save time and resources when designing a study.  That is precisely what King was discussing.  Once the data are collected, it is a mistake to discard some of the data.
Nothing can be "statistically significant" without a context and an objective.  What decisions do you intend to make with these numbers?  What is the nature of your "evaluation"?
Finally right. I can't do LaTex in 5 minutes! As best as I can tell the above equation comes out both ways.
I've researched the idea further and have found that the priors should incorporate a true known effect if they are to be informative. In other words they should be based on some past experiment, or pilot study which lends credence to their assumption. Otherwise I could use a community of priors with them being centered at a null effect of 0%.
+1 concise, clear, good advice. In respect of the last paragraph, I'd agree, except that in smaller samples, confidence intervals will also rely somewhat on near-normality. In larger samples (whose size will depend on  the distribution), yes, only prediction intervals will really depend on normality.
@DavidMarx Typically yes, the only active processes on the test machine should be the performance test, however there may still be some background activity from the OS.
Two very different looking formulas for the same thing. Yours from ANOVA type reasoning, mine from regression.
it would be a pleasure. I will edit the question with a detailed explanation later this afternoon, but in few words is a model of diagnosis of a gene variant by a medical test.
1) sample 1 and 2 are people that subscribed to my service in different country.  2) the 3 measurments are weight over 3 days without treatements 3) they are not the same personnes 4) there is the same number of individual in each sample and if they come from the same population or generated from the same PDF then i guess y for every individual in sample1 you should be able to find a similar individual in sample2
Despite the comment I posted to the question, I have since realized that **the conclusion is not true** except in special cases. The problem occurs with composite hypotheses, such as $\\mu_1 \\le \\mu_2$. "The null hypothesis is true" now covers many possibilities, such as the case $\\mu_1 = \\mu_2 - 10^6$. In such a case, the p-values will *not* be uniformly distributed. I suspect one could manufacture (somewhat artificial) situations in which, no matter what element of the null hypothesis holds, the distribution of p-values would never be anywhere near uniform.
The belief that "The use of the CLT for justifying the use of the Gaussian distribution is a common fallacy because the CLT is applied to the sample mean" is itself a fallacy. For example, the electrons in a conductor are moving about at random. The _small_ charge on each electron contributes to a net _noise_ voltage (called thermal noise) that can be measured across the terminals of the conductor. Each contribution is small, there are many electrons, and so via the CLT, the noise is _modeled_ as a Gaussian random process. This model has been cross-validated in numerous experimental studies.
Do you mean the arcsin(square root(x))?  If not, that could be why it didn't work...
I think Stata does not just divide.  If you are curious, you would use the aweight option with length as the weight variable, but this is not it.  Did you use exposure() or offset() to specify the offset?
@Jawad It would better if you could register your account here on CV (and maybe on SO) so that your comments and future interaction on this site could be linked to your actual question.
Huge thanks; very appreciative! Instinctively I knew that, Peter but darn it anyway...looks pretty good using linear regression :-)  I have recoded the raw scores, deciding that directionality probably isn't as important in the prediction but don't get great results. Important to step back and re-group rather than "fish" for the analysis that "looks good" Cheers :-)
@AlecosPapadopoulos  I want to be able to incorporate the information of the levels of my non-stationary predictors without necessarily including them directly as regressors, which would lead to a biasing my coefficient estimates. I am asking what is the best way to deal with this - the best way may include throwing them away.
If you're able to identify what particular things you need to know to understand those answers and ask about those, perhaps in a new question, you might get somewhere. I'm not sure what else to suggest; CV isn't really an easy place to learn basic statistics, but mostly to overcome specific issues.
Thanks to you too. I think I will need to go away and digest this answer a bit, I really appreciate it though as I think I'll learn a few new tricks with this.
Of possible interest: [Which pseudo-R2 measure is the one to report for logistic regression (Cox & Snell or Nagelkerke)?](http://stats.stackexchange.com/q/3559/930)
Question: I am not familiar with the R-code. Can you write the regression equation specification in mathematical symbols, so that I can understand whether you are running a linear or non-linear regression?
Fantastic, I had to sligtly modify your code because cobs was providing some warning messages on the number of knots.                  co <- cobs(x,y,nknots = 20, knots.add = TRUE)
plot(co)
maybe 'extremes'?
Also, thanks for running a timing test!  I assume by PLS you mean "partial least squares."  Were you able to get 50 components to span the column space of your matrix?  Does this somehow model a function from $\\mathbb{R}^{50}$ to $\\mathbb{R}$?  I would think you would need to matrices for this.

I appreciate your patience!  As long as the learning step takes reasonable time, I'm happy with it -- the evaluation is what must go very quickly (will be running it millions of times). (@cbeleites for this comment; @fabee for the last -- it won't let me edit it again!)
@chl, these are the arrival times of the packets measured in our network. the two dataset is not big, but the sizes are different as mentioned. i just want to show the differences of these time measurements in the same plot. Thanks.
@Dirk Eddelbuettel: Deleted my other post. I wasn't sure if this comes under programming or pure statistics.. Anycase, thanks for pointing it out.
It might help just to change your point of view. It strikes me that this is not a *classification* problem, but rather it is an *attribution* problem. The distinction is that each phrase or sentence no longer needs to be put into one category, but rather can be assigned multiple attributes. From this point of view the problem turns into a collection of (practically) independent problems: you need a job description classifier, a salary identifier, and so on, each of which could be developed and run independently of each other (and even use different methods).
It's just qqPlot() from {qualityTools} :)
I'm somewhat familiar with recommender systems, but my question is more of the form "who is A's favorite director", not "what movie should I tell A to watch next?" Are you suggesting that I build some kind of model, and then presumably that model has some director-level parameters that could be used to say something like "which director has the greatest impact on the probability of A liking a movie?" If that's the case though, then how does it take into account the number of movies A has chosen to watch from the same director, independent of A's ratings on those movies?
This is just a check of the definitions. They have equal distributions. Random variables which take the values $0$ and $1$ are parametrized by the probability with which they take the value $1$. These are called Bernoulli random variables, not binomial random variables. Binomial random variables are something different, the number of heads when you flip several identical and independent coins with a fixed probability of heads.
If you have only rows and columns (a 2-way table), then it is called simple correspondence analysis, not multiple
A typical value is not the mean or median. I said to simulate a future value. Then you are taking into account the forecast variance.
Thanks for the confirmation! I am trying to compare the importance, and the possible interaction, of environmental variables to explain species distribution, using several species as the response variables (you could think of these as my reps). The species all have different numbers of presences, and different relationships with the predictive variables. Could you suggest the "best" metrics of effect size to do this? Area under the ROC curve, % deviance explained? None of these models are very complex I am only considering two variables at a time, with possibly cubic terms and interactions.
@Germaniawerks: In ESL, some of the methods, e.g. LASSO, ridge regression, involve shrinkage, so changing the predictors' scale will affect the model's fit.
Thanks for providing this quote. The original quote can be found [here](http://cran.r-project.org/web/packages/penalized/vignettes/penalized.pdf) on page 18.
lengths of x and y have to be same. If for one x you have multiple y, you may replicate same value of x. for example `x <- c(1,1,1)` and `y <- c(20,32,45)` so both vectors have same lengths.
"I would like to create a measure to find the clusters which discriminate A from B." Obviously and plainly, it is the ratio between numbers of As and Bs in a cluster. If As or Bs prevail in a cluster then that cluster discriminates them well.

Your idea of ratio of mean similarity between As in a cluster to mean similarity of all points in the cluster is questionable. What if you have just 2 As (very close to each other) and 20 Bs in a cluster. The ratio will be >1. What does it tell you?
This blog post by A Gelman discusses the approx. relationship between probit and logit coefficients ["Take logit coefficients and divide by approximately 1.6 to get probit coefficients"](http://andrewgelman.com/2006/06/take_logit_coef/). So, this is one way to go: probit $\\rightarrow$ logit $\\rightarrow$ r
Because not all association has to be related to variation.  The intuition expressed by the O.P. is that when $X$ and $Y$ are both consistently at the high end of established ranges, they are associated.  In this sense, even when there's no variation within $X$ or $Y$, one could say they are simultaneously high.  It is potentially confusing to call this "correlation," which it's not, but it sounds like a valid concept to me.
I have tried to edit your question so as to clarify and to eliminate some opportunities for misunderstanding.  If I have not kept your original meaning, please rollback my edits.
I would avoid Greene *for undergraduates* unless they are bright.
Great link will read it
If my understanding is correct, i.e. my own weights can be supplied thru "weights", and the IRLS itself maintains a set of weights which is "w", and the final actually used weights are the product of my weights and the converged/output "w" weights. Then I am looking for the calculation of the "w"'s at each iteration. The reason is I am actually solving a problem and I casted it into a "rlm" problem and used "rlm" to solve it, but I have to make sure its weighting scheme in IRLS is the one what I intended. Thank you!
Since the $X$s are vectors, how are you defining $X^2$ exactly - element-wise or $X'X$, or...?
The fact that you can refer to a tag shows that similar questions on transformations have often been asked before. Are you sure that none of them address your question? As it happens, the emphasis here is pretty much the wrong way round, as normality of marginal or even conditional distribution is just about the least important assumption which might be more nearly satisfied after a transformation: approximate additivity, linearity and roughly constant variances are all more important.
Could you please give an example of such an alternative model?
Thanks. That was my initial plan. I just wasn't sure if I could just choose 'interesting' pairs of variables instead of going through all the different pairs.
In this case the experiment was done in 5 different centers and 60 pupils were randomly subdivided into control and experimental groups (30 each). Would this be considered a block design? or would have the the 60 pupils be required to be grouped in some way before randomly selecting them and assigning to the experimental/control groups?
@Nick: thanks for editing! Avitus