tttjjj/elt_bench huggingface dataset:
+1 The reduction in parameters also means that ordinal models can be much easier to fit.
Thank you Dan. I will try to convince my university to buy it :)
I'm glad you answered this, giving me the opportunity to add that faceting has a specific purpose that is designed around *identical axes*. The more you want each panel to be designed differently, the more you need a tool that just isn't faceting, and is more like grid.arrange or using grid.layout directly as you mention.
Is your first example on the scatter plot a binary classification? If you use scatter plot in a binary classification, won't you get just two horizontal line?
Hi @Dikran. Thanks for the answer.
Could you please be more specific about "adding an extra input feature..."? 

My thinking:

Suppose that we have already an "SVMs-without-bias" solver $F(X,y,C)$ (that returns $w$ and $\\xi$).

Now for an instance $(X,y,C)$, where the data $(X,y)$ are linearly separable by a hyperplane that does not pass through the origin, if one applies a suitable rotation to the data to obtain $X',y'$ (i.e. we go to another space, the same idea as "feature space" in kernels), then the above solver can be used to classify the new (rotated) features: $F(X',y',C)$.
A. training and feature selection happen on the same test. Feature selection is conceptually thought of as the first part of model training and not a separate process. B. The benefit of bootstrap (or any cross validation approach) is that you can use the whole dataset for model training and testing. So if you bootstrap you don't need to split your dataset at all. (3) Sorry about the last paragraph - almost never use matlab or SMV - so don't think I can add anything useful.
Thank you for asking this as a new question. I assume you are referring to the question [dynamic-conditional-correlation-in-stata](http://stats.stackexchange.com/q/41282/7290), but I'm not sure what table there you are referring to. Can you add more information as @NickCox requests?
It can be called. It's just that outside of the machine learning domain, people don't bother to call every unsupervised method "unsupervised learning". It's unsupervised, but there is not a lot "learning" going on, unless you call any use of statistics "learning".
Likelihood ratio tests are a *class* of test where the test statistic is formed from the ratio of the likelihood under the null to the likelihood under a specified alternative, or more generally, the ratio of the maximum likelihood under the null to the maximum likeihood under a specified alternative. (Sometimes this ratio is a function of a simple sample statistic, e.g. the sum of observations, & so any test based on this statistic gets called an LRT, as it gives the same answers, even though the likelihood ratio may not be explicitly calculated.) So is there a particular LRT you had in mind?
do you have anything in the way of training data? i.e. some data for which you know what the correct answer is?
It is about few lines of code in any decent programming language, there is no need of any library
Since none of your models appear to allow for an error term your choice should be totally straightforward. Likely at most one model will fit exactly, and most probably none will - with no errors, only exact fits 'fit the model' at all. A more interesting question would result after either adding an error term, or writing the right hand sides as parameters in some distribution for $y$, or failing that, at least putting $\\operatorname{E}()$'s on the $y$ variables.
Some more details would be nice, e.g.: what question are you trying to answer? Broadly speaking, cointegration deals with long run relationships and, at least to my experience, RDD is used to estimate treatment effects, so without further information it's hard to answer your question.
The main reason I think is that my modification may also affect distant genomic regions (some trans effect, I imagine if I hit a transcription factor, many genes elsewhere may be messed up). So while I expect local effect, I cannot exclude distal ones... Also but less importantly, I think it would make my statement about local effect much stronger if I could show that only that region is affected in the whole genome.
This problem 27.21 refers back to problems 26.1 and 26.29. I have added those to your question.
This looks rather like bookwork. If this is for some subject, would please add the self-study tag? What do you mean by "the variance error"? Are you referring to the standard error of the sample variance, the standard error of the sample mean, or to some other quantity? Can you be more explicit about how those numbers were calculated?
Let's do a back-of-the-napkin reality check. Three deaths per year is a mean rate of 1/4 per month, so--assuming independence--five deaths in a *given* month has a chance of (1/4)^5 = 1/1024. I would therefore *expect* there to be *some* month in which five deaths occur in any record comprising 1024/2 = 512 months or more, which is about a 40 year period *for a single city (population)*. The news organization probably monitors statistics from dozens if not hundreds of cities; if so, this event is unremarkable. None of these calculations support a number like $10^{-56}$: that's a mistake.
*Inference* is definitely what I'm after here. Anyone can calculate means, and that's usually all that is done. Say, Player A shoots 51% and Player B shoots 49%, you will always see a sports commentator saying Player A > Player B. But given the amount of variation and noise, who really knows if Player A > Player B next year? That's what I want to know. What can inferences/predictions can I really make about these data?
Sounds good @peter, thanks for the suggestion. I'll give it a go, and report back.
When you say that the second dataset is not in any way related to the first, do you mean that the datasets have different covariates, or that they're arising from entirely different situations?
Marc, your present question is purely algebraic: namely, whether one thing (an SD) can be computed from the values of two other quantities (a mean and a CI). On the other hand, the comment to which you refer concerns *sampling*; specifically, predicting results of a second sample based on a first.  There is no useful connection between the two sets of ideas.
Whatever they do, users get points for the amount of activity they do on the site. The goal is to determine how active people are on my site. But I don't know what the maximum number of points to give so I cannot determine an average number of points for each person without factoring other people's points. I want to give each person a score that reflects their activity on the site as compared to others. So if the person gets a score of 90%, that should mean that they are exceptionally active. How best should I look at this situation to determine a score in statistical terms that makes sense?
@whuber: Actually, I think your first concern doesn't apply in this situation because once the device fails there can no longer be any more successes *for that test* because the device hard locks and requires me to cycle the power to bring it back to a normal state.  There is still the concern about me censoring the data, however.
Wouldn't this be equivalent to `lm(price~factor(time)*factor(symbol), mydata)`?
This question may be relevant: http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros
What about a beta regression?
+1 The rules and tutorials look useful, comprehensive--and forbidding!  How is it possible to write about graphics without providing *a single illustration*? :-)
than my answer above fully answers your questions: 'no you cannot directly compare $D_x$ to $D_y$', 'yes, because there is a dimension effect' and 'to make $D_x$ and $D_y$ comparable simply divide them by $\\sqrt{p_x(n_x−1)}$ and $\\sqrt{p_y(n_y−1)}$ respectively, where $p_x$ ($p_y$) is the length of $x$ ($y$) and $n_x$ ($n_y$) the number of elements in $H_x$ ($H_y$).
A hint is provided at http://stats.stackexchange.com/questions/26924.  In general, the sampling error of a *variance* can be computed in terms of the first *four* moments of the distribution and therefore the sampling error of the SD can at least be estimated from those moments.
:Seb The actual data would be more useful which would allow a comparative review of the simple but possibly wrong approach versus the complex but possibly needed model. Complex methods implemented correctly should be able to reduce/step-down to the trivial/simple method. Models should be complex enough but not too complex !
I've tried that series before, and that guy although he's awesome; he umms and umms and umms and umms a lot. It is really distracting
A detailed answer the same question is given here, and points to R packages: [How to correlate two time series with gaps and different time bases?](http://stats.stackexchange.com/questions/46891/correlation-with-different-vector-length-same-time-period)
Also note that a polynomial expansion of a multi-variable function necessarily includes cross products between the variables so it doesn't necessarily make sense to _a priori_ exclude these cross products from consideration any more than it makes sense to exclude certain variables _a priori_. To be fair, I think this question is too vague in its present form to answer (the biggest question is _why_ is the OP including polynomial terms???) but since you chose to answer it, I'm bringing these issues to you.
I don't understand your notation $\\rho_y$ and $\\rho_y$. The formula you give should probably be meant $\\beta_x = b_x(\\sigma_x/\\sigma_y)$. This formula of converting _b_ in _beta_ is valid also when there is no intercept; but then $\\sigma_x$ and $\\sigma_y$ must be standard deviations not from the _means_ but from 0, - they are _root mean squares_ then.
$v_n$ should be 1 in your case. So if the "df" listed by Statistica is greater than 1, then it refers to $v_d$. As for your second question: no.
Check your expression carefully. Should $i$ start from $0$? Doesn't it make the whole expression meaningless?
@fmark, how easy it is to make it work correctly depends on the optimization problem. I use it a lot for teaching with fairly simple problems (one to three parameters) and "nice" models and for simple practical problems. However, if the likelihood is challenging to optimize and profile, it will probably not work out of the box.
These threads might be helpful: **1** [R's lmer cheat sheet](http://stats.stackexchange.com/questions/13166/); **2** [Interpreting three forms of a "mixed model"](http://stats.stackexchange.com/questions/38370/); **3** [Questions about how random effects are specified in mer](http://stats.stackexchange.com/questions/31569/); **4** [Questions about specifying linear mixed models in R for repeated measures data with additional nesting structure](http://stats.stackexchange.com/questions/34238/). (Nonetheless, I think this is sufficiently distinct that it's not a duplicate.)
And now there's [Rack](http://jeffreyhorner.tumblr.com/post/4723187316/introducing-rack).
This is not entirely accurate because you mix up two different kinds of "inference". EM is an algorithm for estimation of unknown parameters, Viterbi is the algorithm for computing the most probable sequence of hidden states. You would, indeed, use EM for HMMs for parameter estimation. I have given more details on the EM-algorithm with historical references explaining the relation between HMMs and EM in my answer.
+1 This is a great answer. However, I think it's incomplete in an important way: in many cases the right mean to use is determined by *the question we are trying to answer* rather than by any mathematical structure in the data. A good example of this occurs in environmental risk assessment: regulatory authorities want to estimate a population's total exposure to contaminants over time. This *requires* an appropriately weighted arithmetic mean, even though environmental concentration data usually have a *multiplicative* structure. The geometric mean would be the wrong estimator or estimand.
Great post!!! thank you for this!
I will read these over the weekend. Thanks!
Please investigate some of the previous threads on the topic of removing outliers: you can find them by linking through the [tag:outlier] tag.
@whuber: yeah, I understand what PDF is - that was just mistake in my description. Thanks for a valid note!
What makes you feel that the answer is not correct?  By the way, if you would like us to check it, please don't make us guess what you're doing: explain what the numbers "2.36", "11", etc. correspond to.  BTW, I notice that some of your previous questions, such as http://stats.stackexchange.com/questions/43197, have been marked up in $\\TeX$: it would be nice if you would mark this one up, too, to make it readable.
Common courtesy is to denote that you've cross-posted at another SE site. http://stackoverflow.com/questions/7291935/how-to-create-a-dataset-with-conditional-probability
This looks like a promising approach when (a) the onset of each signal can be reliably identified and (b) all signals from each individual have common durations and amplitudes.  Although (a) looks likely--the acceleration starts suddenly after a long quiet period--(b) does not look plausible. For good results, some initial processing seems required in order to obtain reliable comparisons among separate signals so that (1) all signals by one individual are as alike as possible while (2) maximizing the differences between signals from two separate individuals.
thank u so so much
@whuber With settlements of smaller sizes (like our simulation output) are included, the lognormal distribution provides
better fit than the Pareto model(Pumain, 1982) > [src](http://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CD4QFjAB&url=http%3A%2F%2Fwww.santafe.edu%2Fmedia%2Fworkingpapers%2F04-02-002.pdf&ei=xb-9UKC9BJO2hAfgg4DYAw&usg=AFQjCNG57tTF7Emu6_5ajKziL9p1cZsLpg&cad=rja).Most study in geography refer to the slope of the Pareto law adjusted to the population size distribution according to the so-called «rank size rule» by Zipf > [src](http://cybergeo.revues.org/3790)
@ilhan Please, consider updating your question (as you did) and point to the updated version rather than leaving such non-constructive comments. Everything is debatable, especially when the question itself lacks some context. Here it seems that the problem stands from defining what a population really is.
What do you mean that your "dependent variable is a probability"? W/ multinomial LR, the DV is a nominal category w/ >2 possibilities.
I think the point is the same as [here][1]\\n\\n\\n  [1]: http://stats.stackexchange.com/questions/97777/variablity-in-cv-glmnet-results/103144#103144
paste the output of `dput(data1)`. If the data is secret, try something like `dput(sapply(data1, jitter))`
If you take the definition of the Pearson correlation and manipulate it so it refers to counts rather than to sums of the differences between individual observations and the means, you get the Matthews formula.  I haven't actually done this, but it must be reasonably straightforward.
Also check out this answer: http://stats.stackexchange.com/questions/69291/which-k-fold-cross-validation-strategy-is-better
you can take a look of the software visumap that enables you to embed high dimensional data into 2d or 3d manifolds (like sphere, projective plane, S3, etc.) and provides interface for interactive exploration.
What statistical language do you use?
@Jeromy: Seen already. The replies and links there seem not formal enough, and too localized to some particular examples.
@Erik If truly have *forgotten* the techniques, then the thing for your to do is to re-read your class/text notes. The solution suggested above does not apply to forecasting forward prices of stocks at all.
+1 Excellent answer.
Delete your question on stackoverflow, and create a new one here.
**Hint**: $F(x) := \\int_0^x \\mathrm dF(t)$.
@PeterFlom Actually I didn't explain that very precisely. My professor adds the $\\delta$'s depending on the topic. Here it happened to be panel cross sectional data and she used the $\\delta$ for the year as a dummy variable (because it was either observed in 2000 or 2010. And anything associated with the $\\delta$ she also uses $\\delta$ as a coefficient. Not that this is the most important thing, I just wanted to be more correct.
@cardinal I disagree very much. This is basically the exact view espoused by E.T. Jaynes in his 2003 book, which is a decidedly un-Laplacian view in favor of a much more objective Bayesian view.
The definition you supply in the question doesn't appear to be correct. That would make quasilikelihood and GEE not a form of M estimation which isn't the case.
Excellent advice Max. Many thanks. Mea culpa for not properly realizing what is required and what is particular to the example.
@cardinal yes I see, thank you for the hint. One can transform $N(0,\\sigma^2)$ to standard normal distribution and use $||ax-ay||=a||x-y||$
First, linear regression does not assume that the independent variables or their interaction are normally distributed. The heteroscedasticity may be a problem. [Here](http://stats.stackexchange.com/questions/175/how-should-outliers-be-dealt-with-in-linear-regression-analysis) is a good post about how to deal with outliers. [This post](http://stats.stackexchange.com/questions/23887/linear-model-heteroscedasticity) may be informative regarding the heteroscedasticity. One proposal is to use robust standard errors or to respecify the model (or transform he variables).
Clasifiers performs a supervised learning: they use the entire data and the class labels associated to each data-point. Clustering methods performs unsupervised learning: they use the entire data without having the class labels associated to each data-point. Now you didn't really answered what is "supervised clustering", it seems that for you it is like clustering, however, clustering is unsupervised ! Then, my understanding is that a supervised clustering method is actually a classifier since it performs supervised learning.
I received confirmation from the package author that this is indeed the properly method to use to deal with series of differing lengths (to obtain a distance matrix).
I'm a little lost. Being sensed is a binary outcome, but the response of the sensors is continuous, and I do not have an a priori threshold. I'm not sure I am comfortable interpolating a two dimensional surface from two one dimensional data sets. I think I was more curious if there were existing models for fusing data from one dimensional functions.
In the latter case we'd need to know what percent of times A,B and C are the right decisions, I think.
(+1) Learn to work with data. A very nice supplement is [Problem Solving: A statistician's guide by Chatfield](http://www.amazon.com/dp/0412606305/)
I think I am looking at the wrong thing alright.  This is a distribution of user activity in system we have.  Each observation is a users activity in the last month.  So user 1 submitted one post while user 781 submitted 23.  I want to present simple ratios like 50% of users contributed 50% of posts (similar to pareto's rule (80/20) which is not observable here).  It helps to give a simple and understandable breakdown of activity but I think I am getting a little confused between approaches.
It is ill-posed density deconvolution problem. Can you provide some issues related with the solution of such problem in real? btw: have you ever seen any tools can help us decompose the estimated pdf to theoretical basis distribution? @julienstirnemann
I'm afraid I dont know how else to qualify it. It is the uncertainty in the measurement, propagated from errors.
Please edit your question to make that clear
Ah, OK.  I should have seen that.
Dilip is 100% right. The fact that the gamma distribution has two common parameterizations is a source of confusion. I'm using the definition (Schervish, "Theory of Statistics"): $Y\\sim\\textrm{Gamma}(a,b)$ iff it has density $f_Y(y)=\\frac{b^a}{\\Gamma(a)}y^{a-1} e^{-by} I_{[0,\\infty)}(y)$. So, $Z\\sim \\textrm{Exp}(\\lambda)\\sim \\textrm{Gamma}(1,\\lambda)$.
...Sorry, that last part is a bit over my head. Do you have a resource I could go to in order to learn that?
Can you please be more specific about what kind of data set do you have? Answers may vary accordingly. `rpart` may be Ok - but may be something better is available.
You sound pretty motivated to get as much feedback as you can, so perhaps you could post plots on a site such as dropzone and give us a link to that.  If you can, I'd first select a random subset of a few hundred cases, then plot regression residuals as the Y-axis and predicted Y as the X-axis.  Then I'd show a histogram of the residuals (for this you can include all cases).
@Henric I use this trick frequently, yet user1414's code return this roll with slide 1, not 2, as intended by OP. Check out `(c(0,0,x)+c(0,x,0)+c(x,0,0))/3` to see what I mean (and how does it work). The proper formula would be: `(c(0,0,x)+c(0,x,0)+c(x,0,0))[1:(length(x)-3)*2+1]/3` (we must cut 0-padding at the beginning and select even elements then.
@Frank Harrell, thats a great question, but it is a separate issue than the one at hand. For the moment, let us pretend that the index algorithm is a good way of interpreting variables.
I agree that if an odds ratio is sought as the primary objective, it makes sense to stick with logistic regression, but of course all those assumptions have to be met. I disagree that a probability of class membership is not "naturally" output by tree based models though, these (and GBM and others) absolutely do.
Thanks for the reply Tom. So, for example to update $\\lambda$ I would also need to provide an approximation for $\\phi$. Would something like the following make sense:
1: Start with approximation for $w$, $\\phi$.
2: Estimate $\\lambda$ using the explicit update
3: Estimate $\\phi$ using known $w$ and estimate of $\\lambda$.
4: Approximate $w$ using EP or any other method using the estimated $\\phi$ and $\\lambda$. 
5: Iterate.
I have edited the question to give a background as to why I am doing this
I don't need depth as my main goal was to create the program for comparing both algorithms, need only basic stuff.
I think you may be confused about what supervised learning is. K-means is an unsupervised learning clustering algorithm. Perceptron is a supervised learning classification algorithm that attempts to find a hyperplane that separates negative from positive observations. LDA is a method that can be used for supervised classification but is more commonly used for supervised feature selection. See @AdamO's answer for assumptions of the LDA classifier.
Indeed, yes, full subset selection (aka best subset selection, all subsets regression, etc) along with approximations like stepwise regression lead to a long list of problems (biased estimators, the inference procedures not having the anticipated properties, that kind of thing). Many questions here discuss the issue. I will try to find some. For starters see [here](http://stats.stackexchange.com/questions/74956/does-full-subset-selection-regression-model-building-suffer-from-the-same-handic) and [here](http://stats.stackexchange.com/a/11276/805).
I have initially imprecisely stated the problem -- it is about finding patterns in a Markov chain given by a transition matrix, and not about storing the matrix in a compact form (although the latter might be a positive side effect).
(Note that it doesn't matter that the principal vectors are determined only up to sign. In fact, it doesn't even matter that the $\\sum_ja_{ij}X_j$ are principal components: the same calculation applies no matter what the $a_{ij}$ happen to be, provided their computation is not based on any values of the *dependent* variable in the regression.)
What do you mean themes of variables? This question might be better on CrossValidated StackExchange.
Would you give the same advice in the second paragraph if the OP indicated that the interaction was not expected to occur theoretically but was included in the model as a goodness of fit test?
I am lost.  How do I calculate a1, c1, and a2?  Once I predict Yt, how do I calculate the prediction interval?