tttjjj/elt_bench huggingface dataset:
Paring comes from the experimental design; for example (1) when the same quantity is measured with two different devices (2) When a certain feature of two subjects is measured together with the same device (this is common in genetics and cytometry). Having the same sample size does not necessarily imply pairing.
I've currently made some notes on my first question, and they can be found [here](https://docs.google.com/document/pub?id=1T0X_HXxC46tJS6y4BcLJlXuXf76x7FReb6snJ1YO4Gg). I'm not sure at the moment whether I should just post this as an answer to my own question or open a seperate question (as this is largely focused on data visualization). But feel free to leave me a comment about the google document either here or in the [chat room](http://chat.stackexchange.com/rooms/18/ten-fold).
-1, what you describe as estimation is simply forecasting and forecasting is not estimation. You can forecast with either estimated or calibrated model.
Does the Gibbs sampler converge to a global maximum in the presence of multiple modes? For example in case of a Gaussian mixture distribution?
@Marco, yes I had only meant to address the issue of if there is a relationship or not.  Estimating the magnitude or understanding he interaction needs a well fitted model otherwise it doesn't make sense.  I'll ensure the answer addresses this
Have you considered the possibility of using both variables simultaneously to predict $Y$?  Incidentally, the (raw) slope coefficients tell you nothing about predictability: after all, if you merely change the units of measurement of one of the independent variables, the corresponding slope will also change: but a change of units of measure obviously will not change your ability to predict the dependent variable!
+1 but I think the "more difficult" way is actually in the end simpler and far better.  One of the main uses of regression is to answer this sort of question.
+1 I think I would use a different adjective than efficiency though (or be more clear you mean computational or code golf efficiency). whuber's comment points to a resource that gives exact intervals, and the glm approach is based on asymptotic results as well. (It is more general though, so I like recommending that approach as well.)
I don't get -- if you have the probability, you simply use the top one, and then you check if the significance is high enough for your needs (i.e. how sure you would like to be the classification is OK).
So part 2 has the Baum-Welch parameter estimation stuff?  around what minute mark is that?  Thanks
I would venture to guess that if the race is popular, any wild animals will be long gone well before hordes of people descend upon them, suggesting the *real* answers are "are you kidding" (to (a)) and "practically 100%" (to (b)). :-)  Behind the facetiousness is the quite serious point that in many situations, authoritative quantitative assertions about probabilities only obscure a complete lack of applicable information, so we should develop the reflex of questioning the basis of such assertions before we ever proceed to do any calculations.
Related: [Computing percentile rank in R](http://stats.stackexchange.com/q/11924/930).
Very good, thank you. Also $e^x=\\sum_{i=0}^{\\infty} \\frac{x^n}{n!}$ ;)
I added the extra info under the edit
It isn't a matter of shooting you down, I don't think that there CAN be a solution to these problems.  We are humans recognizing patterns in the world, we have to settle with convergent validity.  After his poor attempt at proving a god exists, Descartes bunted to convergent validity.  Sometimes it's there, sometimes it isn't, but we mostly just run up against our infinitesimally small powers of cognitive computation.
Do the two methods yield the same number of clusters w/ the same number of objects in each cluster? If these are different, can you specify how so?
If you are doing multiple testing and want to control the overall (so-called fmailywise significance level) you should adjust.  Bonferroni is one conservative way but is not the only way that you can make the adjustment.  If you are just using an informal procedure for deciding when to remove a variable from the model then you do not.  But oyu don't have to use 0.05 either.  people sometimes use 0.1 or 0.2.
I would suggest that you give _some_ thought to your choice of notation. Given that A, B, and C are already used as name brands and you have decided to use A to mean Brand A is chosen, you are just setting yourself for confusion by choosing B to denote Jeans cost over \\$100 instead of B being chosen. Couldn't you have thought of something else? Say O for over  \\$100 the way that both answers did?
If the answer is helpful you might consider accepting it. If not, then you might say what is still unclear.
Other than little men (cows, beer barrels, sheaves of wheat), the generic term I know is pictogram. (I am not sure I understand "color coated" here; the objects can be black in colour. Do you mean "coded"? Still don't know what that would mean.)
(1) No. They are not using cross-validation. See the Hansen article.
(2) Use approx().
I think that was what I read about. Really neat :)
On train/test/validation. Good practice, but cross-validation (on train) then test is just as good. A third, separate dataset is often needed when comparing the performance of different types of models, but here you're looking at two models of the same type, I assume.
Have you looked at whether another type of distribution matches your data, e.g. Poisson or binomial? I'm assuming from your question you've noticed that your data is not normally distributed and haven't investigated alternative distributions.
I was the one who originally posted this question.   Thank you for your response.  It makes some sense!  I appreciate it. As a matter of fact,mu  *does* take on a set of discrete values!
An output of "str(data.ex2)" would be helpful. This is because if "subject", "concentration" and "vessel" are not factors, aov will give strange numbers (particularly subject and concentration in this instance).
I wouldn't be satisfied with a confidence level as low as 95% or even 99%; extraordinary claims tend to require extraordinary evidence.  Or to put it in a more Bayesian context, my prior belief of him having such an ability is so low that I would require a ridiculous amount of evidence to meaningfully alter my posterior belief.
@dissertationhelp: I have just updated my answer with an important remark, please take a look.
@colinfang I'm afraid you need to look at the source code to check that, as this is probably non-standard (Personally it should throw an error, not a warning).
How exactly do you fit the logistic regression to get standard errors? If I do maximum likelihood estimation, I will only get a point estimate with no standard errors. If I do Bayesian, I will get that; but I don't know if there is work on Bayesian Adaboost.
It sounds like you are quoting something.  Could you provide a reference or link to the source?
Sorry! I didn't see you responded. I'm not sure, I have never tried to do that. You could just post it like code using 4 spaces at the beginning when you write it so it will have a scroll bar. Or just post on different lines.
So you measured the fitness level 4 times? Your first comment suggested 3 measurements: Baseline, 4 weeks, 6 weeks. So I got confused. You are sure that fitness level is metric, e.g. data from a bicycle ergometer? And: Have a look at [this](http://stats.stackexchange.com/questions/13197/differences-between-manova-and-repeated-measures-anova/99060#99060) older question, please.
See e.g. [Pewsey et al. (2013), *Circular Statistics in R*](http://ukcatalogue.oup.com/product/9780199671137.do) & the R package [circular](http://cran.r-project.org/web/packages/circular/index.html).
so, if not to mention the sited above, what is the difference in your view?
Not sure I understand your question. The ranges #### - #### were provided. Since they are not numbers, they are treated as factors during the linear regression i.e. as.factor
Not quite: I am very prepared to learn that the term might  be useful, but that is not my experience. Your last question is a good one; my own answer would be phrased in terms in the sociology of knowledge, i.e. that most people just copy out of texts and some people try to understand them.
A(i,purple) will be high, so P'(i,red) will be lower than if purple was further away.
As I also said before, this error indicates that you have not fit the model using the training set. You can't evaluate it on test_set without fitting the model first using either fit or fit_transform. Please see my answer and the examples in the documentation of scikit-learn.
EP does seem cool (although it still hurts my head).  Does it still lack general convergence guarantees?
Thanks, yes I have missing data, but this will not affect the whole country; this only affects the company in that country; and thus, this company will not be included and the other companies in that country "country 9" will be included in the analysis
One note - OLS regression does not require that the *data* be normally distributed, only that the error, as estimated by the residuals, are.
Using 0.5 is a good "default". However, if your two misclassifications have different costs or implications, another number might be better.  What is the context?
"Sample 10% of the population" has almost no basis in statistical theory or practical application.  (In fact, it is contrary to statistical theory, which shows that usually the absolute sample *size* matters, not its percentage of the population.)  More importantly, any such rule is assuming something about the trade-off between the costs of sampling and the value of improved decision making *to you, in your particular circumstance*.  That's *exactly* the sort of thing you don't want a stats cookbook or glib Web site to decide for you!
@mbq: Already add an example in the revised version.
Look up the answer to this question : http://stats.stackexchange.com/questions/72857/derivation-of-conditional-distribution-from-other-two-distributions/72870#72870
Please, take a look at the paper posted by the OP. These terms are interpreted differently there. But I agree with definitions you are giving in the Statistics argot.
My two groups are actually overlapping in term of age. Thank you for your proposition. Just to make sure I get it right, were you referring to logistic regression, with group as the dependent variable and the balance score and age as covariates?
I'm not sure about the wording you have for part (3). 

It should be obvious that both $Z$ and $W$ must take values in $[0,1]$, and so the pair $\\langle Z,W\\rangle$ must take values in the square $[0,1]^2$ (and in fact within a triangle contained in this square).  The joint density is therefore a function from this square to the real line (you can think of it as a surface).  The important point here is that it is a function of TWO variables: $\\langle z,w\\rangle$. 

So, to cut a long story short, I don't know what is meant by "the joint density is [...] 2 for 0".
I think so. Try updating caret and gbm.
With one df the range is not even an approximation--it's directly proportional to the SD :-).
Could you clarify whether the Ms Hudson to whom you refer is Kate Hudson or Jennifer Hudson??
Thank you very much for the link. And I do totally understand that I will need to provide a training set to the algorithm so that it will be able to rank subsequent items.
@Tomek, what are you not getting from this answer? If you are using the randomForest package in R then the plots Zach describes should be very useful. Specifically, you could use varImpPlot for feature selection in your logit model and partialPlot to estimate the type of transformation to try on continuous predictors in the logit model. I would suggest that the latter plot be used to determine where nonlinear relationships between predictor and response exists and then allows you to make that transformation explicitly or to use a spline on that variable.
Thank you for your response. It took me a while to experiment with other data and think about your response to really understand it.
Unfortunately, the snippet above would not help. Making sure that all columns (variables) are present in the data frame is one thing and it would never help against accidental misspelling of a variable name in some other corner of the code.
Thanks. I was actually using Rf_dpois in a C++ program using `Rcpp` and testing it using `inline`. I have been been a user for about a week. But I very happy with the way I can test my C functions with `inline`. It turns out I am not smart enough to see this simple proof of concept. Thanks again!
+1 for NLTK. Olivier Grisel, who is doing Machine Learning research and Natural Language Processing, has some good tutorials and slides around the web. I can think of [Statistical Learning and Text Classification with NLTK and scikit-learn](http://www.pycon.fr/file/2308?vid=download) (PyCON FR 2010), for example.
Hi Gung.  You wrote "If you do drop them, then the subsequent tests are incorrect (e.g., the p-values that are displayed in software output are not the actual p-values)." I know Frank Harrell talks about this alot too.  But isn't it true that, with the nonsig. predictors included, the model is overfitted, such that the parameters for the sig. predictors will get distorted by noise?  And isn't getting the parameters estimated accurately often more important than getting accurate p-values?  Another way to say this is that parsimony will lead to a better model on crossvalidation. No?
l_1 minimisation falls into the framework of linear programing, this can be done by simplex or interior point methods (if the dimension is not too big (i.e. <10^5 or 10^6)
(+1) I would say `points(jitter(rep(1, length(bar$out))), bar$out)` is shorter :-)
@PeterFlom I'm not sure how I should interpret this statistically but in layman's term I think it means the two events A and B do not occur together just by chance.
That helps. Thanks for quick response
Sorry, the lower the score the better. Pooling in each competition was allocated using an individual time trial run before the mass start race. The fastest two from each team went in the "A" race, the next two in "B" etc. The race with only 10 contained the slowest 2 competitors from each team in competition 2. I suspect you're probably right that I've made it a but complex. I do have points breakdowns for all teams in the competition and a quick glance shows the winner of competition 1 to be more consistent, with no winners, but no significant poor performances either. Thanks for your help.
The usual way to accomplish this is to do the PCA on the correlation matrix instead of the covariance matrix.
Well, there is always the possibility that your features do not give you useful informations about your scalar values... Have you tried to do some exploratory analysis of the features? For example, looking if there is correlation between them, comparing the mean value for different subsets, things like that. That can give you a hint about how informative your features are.
The paragraph from pages 104-105 speaks for itself in documenting nonsense.
Everything is just an observation. Reminds me about the anecdote, what is a probability of getting hit by the bus. 50%, you either get hit, or not.
@chl It would be very disappointing if this workaround was the best we could do for such a simple question. In Rcmdr DOE plugin there's an "Inspect Design" submenu under "Design". Seems relevant, but I have no idea how it works.
It's also a *count* - for most analyses you might want to do it's not rescalable.
It's very common for people to make mistakes when they implement learning algorithms. If you use gradient descent, check that a small step in the direction of the gradient actually decreases the cost function by about the amount it should. This is an easy test, but many people skip it and waste time tweaking an algorithm with a sign error, or some similar problem.
Sorry, I meant each individual time series (experimental trial). There is quite a bit of variability among subjects, but the resulting ROC for each test of a single predictor on a trial has a sharp corner like the above picture, just locatedin a slightly different place. I double-checked the size of my predictor and response inputs to the roc() command; both are length 7500 class labels.
Thanks Anne I'll take a look at your references. I had thought about this myself and just realized that this was what the issue is. I'm fairly sure my data has significant autocorrelation. Thanks again.
@Alexis, where are you going with that?  Your comment does not negate anything that I stated.
Nice! What happens internally when this flag is set (i.e. how are the confidences estimated) ?
What is a commonly used / decent / proper prior for the precision parameter $\\tau = \\frac{1}{\\sigma^2}$ if you don't have much info about it? That is, it is non-informative, so to say.
One very basic way to argue this is through "confounding" - both A and B are counting things that are "caused by" clouds.  Hence, in the abscence of data on cloud, they may appear to cause each other.  Is your question "time based"?  By this I mean your hypothesis is something like "hail today means more rain in the (near) future, on average."
+1, nice application of delta method.
Maybe I'm missing something, but could you explain why you choose multinomial regression to model the dependent variable "memory score"? To me, it seems "memory score" is not categorical but can be measured at a fine scale.
Many thanks for you answer, Alecos. ;)
@Alexis in any case i am not bothered about the self study tag . Just wanted to understand the two ;-)
You might find useful clues in this related question, [Multiple Chi-Squared Tests](http://stats.stackexchange.com/questions/1133/multiple-chi-squared-tests). Also, your question seems related to your previous one, [How to carry out multiple post-hoc chi-square tests on a 2 X 3 table?](http://stats.stackexchange.com/questions/9685/how-to-carry-out-multiple-post-hoc-chi-square-tests-on-a-2-x-3-table).
It doesn't mean "you could assume that the *sample* is normally distributed even if the population isn't", it might mean you could assume that the sampling distribution is normally distributed even though the sample (& presumably the population) isn't.
You might consider adding more detail to explain the specific issues of your problem, e.g. size of the state space, sparsity and properties of the transition matrix etc. Otherwise one could simply say that in most cases (with some hypothesis), finding the stationary distribution involves applying powers of the transition matrix to any starting distribution
see below for answer:
http://en.wikipedia.org/wiki/Log-normal_distribution

In contrast, the mean, standard deviation, and variance of the non-logarithmized sample values are respectively denoted m, s.d., and v in this article. The two sets of parameters can be related as (see also #Arithmetic moments below):[3]
Beside "information gain" (in the case that we have two best info gain), what should be another criteria?
I would have thought so, active learning tries to successively identify patterns that produce the greatest improvement in the model.  If all of the unused patterns in the pool are negative patterns, hopefully it will select the most informative ones first.  However, as for most statistical algorithms, your mileage may vary (as they say http://en.wiktionary.org/wiki/your_mileage_may_vary )
I have recently tested tSNE, CCA and RPM algorithms to reduce 1500 dimensional data to one dimension. tSNE worked pretty well, but not the best from a particular point of view (see http://jamesxli.blogspot.ca/2013/11/on-multidimensional-sorting.html). To make a judgement about the performance, I would trust more domain knowledge or visual checking, than ad-hoc statistics.
Did you use one forest per feature, or multivariate trees? Why is being 'too wide' a problem? Did you try feeding this representation into another random forest to get a 'deep' network?
It's an interesting combinatorial problem to maximize $R^2$, but its use for any statistical purposes, such as explanation or prediction, is highly suspect: the resulting model is very likely to be wrong. Much about this issue can be found on this site by searching with keywords related to model fitting, such as "model," "regression," "stepwise," "AIC," and many more.
Your clarification really helps, thank you. It's beginning to sound more like a data structure issue. I agree this is a logistic regression now that you state the outcome as "refer to a specialist". Your variables are as follows: `gp`, `pat`, `pat_age`, `pat_sex`, `cons_x`. Your model is `y ~ gp:pat + pat_age + pat_sex + cons_x + cons_x*pat_sex + cons_x*pat_age + cons_x*pat_age*pat_sex`. You are going to need to collapse your multiple records due to consultations into one record, where the variable `cons_x` indicates the # of consultations (use R to count the # of records per patient).
An estimator is not a parameter, but a random variable. Basically, your estimate depends on the sample which is random, which makes your estimate a realisation of a random variable called estimator. Hope this helps.
Following the comments on Nick's answer: Maybe you want something like [mutual information](http://en.wikipedia.org/wiki/Mutual_information)?
But on the other hand, we cannot conclude independence between the 3 dimensions within each $t$, so I don't think we can assume independence between all dimensions in the big multivariate distribution.
It really depends on the specific problem and data set, sometimes a straight line is the best difficult to say anything without seeing the data.
thank you. that is super helpful.
@mpiktas, The data are indeed survival data in that the dependent var is the time from diagnosis until death. Out of 1500 patients I have 15 that are still alive and hence right censored.
Hi there, could you add some more detail to your question. Could you tell us more about your research? Some questions you should address are: What is your research question? What is your design? Are there 20 measures per person - one control measure matched to one test measure? Is the order of conditions always control followed by test? Is this a biological experiment where there is a known mechanistic process which affects the values - and needs to be taken into account in the model? Is duration important (e.g. time lag between control and test, time during condition to produce measure)?
The result got bad because what you are trying to classify is people (each person will go to some class) and when you modified the data set you duplicated the people hence in a way averaged their weekend and weekday consumptions. This will definitely bring the two classes together instead of separating them as much as possible. And adding the day variable doesn't solve this problem. What you are classifying in the new dataset is the (person, day) pair (the unique key) which is I believe not what you want.
@KateRin can you explain what you mean by "first seasonal difference on 80 and second seasonal difference with 81 pediod, and twice of 81"
We closed the cross-post on Stats.SE due to the duplication, but if you want to migrate this to stats, it would be on topic and we would accept it.  (It already has a good answer here, though.)
yes that's straightforward but is there a way to go backwards and create the predictors from a fixed dataset?
How do permutation to repeat the experiment make a difference in leave one out cross validation, since all points will be tested anyways , why does the order matter?
Welcome to CrossValidated. I think you describe is feasible but I need some clarifications for a full answer. :) Here goes: 1. You fit an individual ME model to each FPC score right? 2. You do not interpreter each FPC individually do you? 3. Do you or don't you have multiple readings for a single individual at your recording times? (I think you do but maybe I misunderstood what you mean) In general the first two sentences of your post are the most ambiguous as the sound a bit too field-specific.
What is the expression for the probability density?