tttjjj/elt_bench huggingface dataset:
Your test may be accomplished by something similar to a matched pair test, unbalanced randomized block design. But before I guess further, could you please elaborate on your data, what is it what is looks like etc.?
+1 A standard way, in matrix notation, to express this $\\text{SSE}$ is $Tr((\\mathbf{A-B})^'(\\mathbf{A-B}))$.
This is a really interesting question. My (relatively uninformed) approach would be to find the most likely state-path in the HMM given the search string. Then, it's easy to find the likelihood of other strings, given that state path. Making it into a full-blown distance metric sounds hard though (Triangle inequality? What?).
(+11) Don't know how I missed your answer!
(+1) A nice answer with good resources; don't be disturbed if this question ultimately ends up closed as off-topic. :)
- and how does these two classes differ from "classics" - songs that were presumeably well-known when they were first uploaded on YouTube (I'm thinking David Bowie)?
$X$ is uniform if that helps.
Something sounds amiss.  Did you randomly divide your data into the two groups of 70 and 30 observations?  If so, I would expect the results to be somewhat similar to the results you got on the full 100 observations.  Are you confident that there is not an error in the way you carried out the lasso or selected the lambda?
["An introduction to ROC analysis"](http://vicos.fri.uni-lj.si/data/vprsistemi/ROCintro.pdf) by Tom Fawcett helped me to better understand ROC curves. You might enjoy it if you are looking for additional literature on the topic.
why don't you google AR(P)? [Wikipedia](http://en.wikipedia.org/wiki/Autoregressive_model#Characteristic_polynomial) article on AR processes has all the answers to your questions. AR(P) process is stationary under some conditions
Since this question is rather vague, the correct answer is: You need at least three trials to achieve one success out of three trials.
Your regression software will take care of numerical issues automatically--in particular, it is highly likely to center and standardize your data internally. How to answer your questions about centering comes down to how you wish to interpret the coefficients.
A *Mathematica* solution is given (and illustrated for the 3D case) at http://mathematica.stackexchange.com/questions/21396/how-to-use-a-33-covariance-matrix-to-plot-an-error-ellipsoid/21417#21417. It recognizes that the contour levels are given by a chi-squared distribution.
The hinge spread (or H-spread) is not identical to the most usual definitions of the IQR in small samples. If you want R to compute the hinge spread specifically (say for a box-plot), you can use $\\hspace{2cm}$  `diff(fivenum(x)[c(2,4)])`. On your data (`diff(fivenum(c(5,8,9,14))[c(2,4)])`) that gives 5.
Links to references would be helpful in getting our heads around this.  Also, please clarify the nature of the covariates & the response variables.  This is a bit too abstract at the moment.
The *first hit* upon Googling the very title of this question produces a good answer: please read our faq about the wisdom of doing a little bit of research before asking a question.
Actually, the point was to predict churn with the purpose of contacting the flagged accounts in an effort to convince them to stay.  Also I am not sure how it would be possible to calculate the probabilities that you mentioned, given that the intervention would be a future event.  Are you talking about using data from past interventions to calculate these probabilities?
Time series modeling is a vast topic in statistics and social science. But, the logical hurdle that you have to clear is to establish that there is, indeed, some underlying systematic component to your data generating process: do you have information about these future months that might influence how this process unfolds? It is possible that the data are just coin flips or dice rolls, implying that attempting to predict them has little value.
You might edit out your answer and put it as an actual answer, so that the question doesn't remain "unanswered".
This is a split that uses more than one variable.  Geometrically this means a splits along a hyperplane, rather than a plane perpendicular to one of the axis.
I think the issue is more that your process creates many models.  In my work (with prediction only), I eventually deploy 1 model.  So even if I use cv to choose among many competing models, I still must fit 1 final model using all the training data.  I can then report the error of this final model on held out test data, and the training error to indicate if there is any overfitting.
Fair points. By "figures" read "no.s". 'Normality assumption' means you assume that when given mean and sd you're assuming the distribution described really is normal. I agree with you about P, although there is a subsequent chain of *inferences* (given the other assumptions) which leads to P(D|H).
When you add the $n$ fractions involved in $S(\\theta)$ (which evidently is the derivative of $l(\\theta)$ w.r.t. $\\theta$, which is the log likelihood for an iid sample of size $n$ from a Cauchy distribution), you obtain a fraction whose numerator is a polynomial of degree $2n-1$ in $\\theta$: that's all there is to this question.
Thanks for the answer, it'll take me a bit to digest it (and I already have plans for my Saturday night). It is different enough that it is not clear to me if it gives the same answer as the GLMM approach, but I need to think about it more.
Thanks. Yes, 3D can be effective, but often it's not obvious what the 3D representation is when it's been projected onto a static 2D computer screen, unless you can grab it and rotate it yourself. I have a 3D version of the above plot, which I'll add to the answer.
But what if we assume, the users just recorded as much data as they can in their free time?
@gmacfarlane spearmens allows for strings to be ranked then correlated another formula can then be used to derive p
Thank you so much Ben! This is exactly what I was looking for.
This is a good answer, but I think what the OP describes is an interval variable under Steven's classification (so no need for another name). If you can apply arithmetic to the original variable certainly you can apply the same arithmetic to the differences.
What quantitative theory relates increased demand to reduced value? Without such a theory, this question seems impossible to answer objectively. Please tell us what this theory is.
Your model doesn't properly capture the time dependence caused by the cycles (even for your simulated case), so your characterization of '*accurately account for*' is not appropriate. The reason you still have pattern in your residuals is probably because of that.
@Macro: OK, I gave it a try. Not sure if it makes the answer better or worse, though.
@ Stéphen Laurent: Could I ask you one more question? How can I prove the diagonal entries of matrix $\\boldsymbol{A}$ are not Gamma random variables if using the different $\\boldsymbol{\\Sigma}$? Must I use the pdf? (It is difficult way, isn't it?)
@Casey Just invert the order of layers and this solution will work as wanted.
Component scores = standardized data * U
Other posters have noted that stationarity is important here. If both series have a linear upwards trend (one kind of nonstationarity), they will be correlated - but all the correlation may be due to the common trend, which may or may not be what we are interested in.
(+1) An equivalent approach (using the same formula)--and therefore more ammunition for justifying this answer--is that you can compute a *prediction interval* for one future value from the treatment group. If the control value does not fall within that prediction interval, you have significant evidence of a difference between the two groups. The difference could be some combination of a difference in mean *or* a difference in variances, but there is (likely) a difference.
1) Sure, it can.  Let H0: X ~ N(0,1), suppose X1,...Xn ~ N(1,1), and let n go to infinity.  That's consistency for the alternative $N(1,1)$.  The KS-test is nonparametric so I bet it would be lots of fun to show consistency for all possible alternatives.  2) Yes, they change on the order of $\\sqrt n$, say.  3) I'm not quite clear on the question(s): are you wanting to show the asymptotic power of the KS-test is 1?  The first step would be to find a definition of asymp power that makes sense for a nonparametric test.  I don't happen to know of one, in Lehmann's book or elsewhere.
@Chris if b>c it does not mean that $x2_{t-1}$ predicts better than $x1_{t-1}$; just that the estimate of that slope is higher.  You can easily have a high estimated slope but very little explanatory power.
@Abe: You will almost certainly not find an "original" reference for this. It is not the sort of standalone "publishable" result found in academic journals. It is simply a (rather tedious) calculation following from the basic properties of mathematical expectation. Quoting a textbook like MGB is perfectly reasonable and acceptable.
Have you have a look at this: http://stats.stackexchange.com/a/32350/7744
It's not "unreliable": it's *wrong*. Specifically, it calculates the p-value assuming the parameters are known; when they are estimated, the p-value as calculated under such a contrafactual assumption has almost no meaning.
@cbeleites: I have a grid of a city. For each grid-cell I have a value saying how many people life within each grid-cell. I am analyzing the spatial autocorrelation of the grid-cells.
We don't. It's a heuristic (and a *very* good one if certain assumptions are approximately met, especially when the data are drawn from a multivariate normal distribution). Finding the global maximum would require searching the entire space of divisions of `n` elements into `k` groups, which is on the order of `n^k`, and is therefore practically impossible.
@rocinante. Given the information stated in the original question, I don't see what else you can do but pick the largest dollars per day. Given more information (sales on each day; reason for why some ads ran longer than others....), you can do more.
Now answered over at: http://math.stackexchange.com/questions/819492/matrix-calculus-derivative-of-scalar-function-of-matrix-function-of-scalar
I believe you may be talking about [this paper](http://www.jstor.org/discover/10.2307/2684348) - D. Edelman (1990)
"A Confidence Interval for the Center of an Unknown Unimodal Distribution Based on a Sample of Size 1", *The American Statistician*, Vol. 44, No. 4, pp. 285-287
How about this: `df[rep(seq_len(nrow(df)), df$count), 1:2]`?
@ David J. Harris See my answer below.
Yeah, it is long and painful to optimize GBM. If you do find a better strategy please share it! In practice I find there is a high variance in the optimal n.iterations which I circumvent somewhat by using Bagging, but it is very computationally expensive.
Hmm, that's a good point (I should read more carefully). However, I can't tell if the OP means that their SE's are high *relative to the coefficients*, or just high in general; the question seems ambiguous on this point.
@joms There is no such thing as being "in" Factor 1:  each person has a score on Factor 1.  Perhaps the men have scores shifted lower compared to women's--that's the sort of thing you could show with the Explore command.  See if you can generate a single boxplot at the end of your output that shows all the Factors split by gender.
(1)  for example, I would like to collect a iid sample $\\{ x_1,\\dots, x_n\\}$ of the stationary distribution, which indicates the sample size is $n$. Is the sampling period same as the sample size?  (2) if I want iid sample, shouldn't I wait a burn-in period between every two sample points $x_i$ and $x_{i+1}$, $i =1,\\dots, n-1$?
Because the model's missing an important predictor.
(+1) Excel actually can do multiple regression and has a command capable of producing a standard summary table.  Given its historical tendency to be (very) sloppy with computing distributional values, the capability has to be viewed like [Samuel Johnson's dog](http://www.samueljohnson.com/dogwalk.html#53): "...a dog's walking on his hind legs. It is not done well; but you are surprised to find it done at all."
This is a fine answer, but IMO over-generalizations aren't very helpful (you can write bad or good code in practically any language, Stata isn't magical at enforcing good coding standards). I have a hard time seeing how 2mb of SPSS code is efficient SPSS code to begin with (with the newer versions with syntax highlighting it would be ill advised to even open such a file in the editor). Seems there is a good chance it was time well spent re-writing in any language.
Have you thought about using a [fractional logit or zero-inflated beta](http://stats.stackexchange.com/a/95589/7071)?
Both the original question and this response use nonstandard notation in respect of the great bulk of the statistical literature. The answer is correct (and +1 for it), but if someone's trying to understand regression by looking it up elsewhere and comparing it with this, the  various mismatches may be confusing. In particular, in statistics, the constant is normally listed first, and the OP's $w$ is normally written as a column vector, not a row vector and would be denoted by $\\hat{\\beta} (= w')$, or sometimes $b$, so that $\\hat{\\beta} = (X'X)^{-1}X'y$. Hopefully this may clarify.
This has been posted on SO and removed, which suggests you are looking for a statistical solution to your problem. It sounds, however, that some pieces of information are missing: What are you trying to do precisely?
In your final decomposition, how are you able to pull the $I_{\\{X\\leq0\\}}$ outside the expected value?  If it's by itself I understand it becomes $P(X\\leq0)$, but in the first one it's multiplied by $\\Lambda(X)$.
I hope this question is not too long. Please give me feedback if it is, and I will split it into sub questions. Thanks in advance!
the two datasets, how many variables do they have (are they each univariate)?
@RussellS.Pierce The edit that contained the question as it stands now - even though it was made before my comment - wasn't there when I loaded it and started researching the original question and possible answers, it only had what I'd describe as a rhetorical question there. The question as it stands now is fine.
Thank you for your answer. That's true that my context is not that clear. I have a sites (rows) x species (columns) matrix with each entry corresponding to the abundance of the species in the site. I would like to investigate the changes in species composition between two years. In the case of a treatment I use the function "adonis" in R for comparing the variance in species composition between the different levels of the treatment. However I don't think that I can do the same for a paired design.
+1 That's certainly a lively way to put it!
You were told nothing of the sort: you were told that when the probability of observing an occurence in a single trial is $\\frac{1}{n}$, the probability of observing zero occurrences in $n$ independent trials tends to $\\frac{1}{\\mathrm{e}}$ as $n$ gets larger; & that an observed frequency *is not the same thing as* an underlying probability.
@Alexander, try [this](http://en.wikipedia.org/wiki/Contrast_%28statistics%29); if that's not what you're asking, ping me again with more info.
No, the jar itself is not the experimental unit. I think of experimental units as rows in a dataset. Your fundamental level of observation is the measurement you take on the jar, for some combination of controlled, uncontrolled, and semi-controlled environmental variable. This is the fundamental concept of repeated measures.
I am not an SPSS expert, but to me it looks like it will be difficult to help you if people do not know what is your data, how you bootstrapped it and what score you measured in the end.
Cammil, a possible interpretation of the question is that you have a large batch of $N$ data values $(x_1,x_2,\\ldots,x_N)$; you stipulate a smallish number $n \\le N$; and you wish to determine (or estimate) the distribution of $(x_{i_1}+x_{i_2}+\\cdots+x_{i_n})$ as the indexes $(i_1,i_2,\\ldots,i_n)$ range over all $n$-element subsets of $1,2,\\ldots,N$. Am I on the mark?
Could you describe what you think your code does?  Apart from the rounding to two decimal places, I cannot see any connection between it and the question (which may largely be the fault of the question, but some clarification would be gladly received).
If gender is treated as a feature in classifier 1, then at some level the SVM algorithm itself solves this issue for you. i.e. gender would be used where and when needed. I know this is not exactly what you're asking, but my 2 cents.
This ends up being a FAQ, see [In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?](http://stats.stackexchange.com/a/3530/1036). Although it is asked about the log transformation instead of the reciprocal, much of the advice applies the same. I've also added the [tag:data-transformation] tag, see the highest voted questions in that tag.
@Peter There are $\\binom{1000-1}{3-1} = 999*998/2$ ternary splits, actually.
I am not sure I understand your question. Here is a sample row of an independent variable after regression analysis from the paper. Transit supply characteristics: Number of bus route (buslines) | 0.275(Beta) | 5.70*** (t-value)
Thank You. I think I get it, but I want to make sure. I the following statement right? If I only have two outcomes, then most information I can obtain is 1 bit, but if I have more than 2 outcomes than I can obtain more than 1 bit of information.
(+1) Thanks for linking with OpenMx! Really great package that has replaced Mx on my Mac now.
Thanks for your helpful answer! I was hoping to find a way to use lme() in order to report p-values. Unfortunately it seems that pvals.fnc() will only work with a lmer() model 'when there is more than one random intercept in the model'(R Documentation). So the only random effects specification I would be left with is the first one you mention: lmer(memory ~ sess*treat + blood + (1|id), data = data). So it seems I have to give up on p-values and use likelihood ratio tests instead?
@Manuel  Thanks.  I caught an error in my R program---there was a 0.04 where there should have been an sqrt(0.04).  It did not change the thrust of the answer.  It made a small difference to the results.  However, if you copied the code before, you should copy it again now.
This cannot be a probability mass function. The total probability mass postulated is $a^2 + a(1-a) + (1-a)^2 = 1-a+a^2$. For this to equal $1$ we must have $a^2=a \\Rightarrow a=1$, which concentrates all probability mass at $f(y=0)$.
Hi @kc2112, welcome to the site. This contribution looks like it could have been posted as a comment. It's helpful to have a little more info in an answer. Would you care to say something about the pros & cons of `glmulti` vs. `glmnet` (which was already proposed). Do you like it better? Is it faster, or more stable, or just preferable under some circumstances but not others? etc.
Hmmmmmm, linguistic pedantry?  Sure... But still about statistics.
Define "beat". :-)
(+1)For your continued interest in this question and your independent follow up. :-)
Does your formula return exactly the same result from the empirical and the theoretical? Mine seem to be proportionate to each other but the actual values are very different.
`PROC LOGISTIC` in SAS produces the usual wald-type tests based on the $z$-score. I wonder what prompted the change in the newer function (byproduct of generalization?).
@ADEpt I don’t know what you mean by range questions.  The specifics behind the model that Akinator.com are… well… specific to them.
You can have a look to coding convention : [Google's R Style Guide](https://google-styleguide.googlecode.com/svn/trunk/Rguide.xml#identifiers) or [Bioconductor - Coding Style](http://www.bioconductor.org/developers/how-to/coding-style/). This does not fully answer your question but it's a good start I think.
Hello, Christelle, and welcome to the site!  I have to say, though, that your question, being about trouble with a specific R package rather than about a statistics problem as such, would probably get a faster answer over on stackoverflow, where I notice quite a few questions with the 'traminer' tag.
Did you consider using Central Limit Theorem? $X_i^m$ will be iid., so $\\sqrt{N}(Y^{(m)}-EX_1^m)\\to N(0,var(X_1^m))$. You can find $EX_1^m$ and $Var(X_1^m)=EX_1^{2m}-(EX_1^m)^2$ by differencing the characteristic function of $X_1$ and evaluating it at zero, which for normal variables is not that hard to do. As a bonus, all the higher moments will be expressed as the functions of the first and second moment.
Your data set really doesn't have enough spatial coverage that you should be estimating autocovariances at a distance of 5 kilometers (the whole area isn't much more than 10 kilometers across and you generally want to have a data set that covers many times the correlation length.)  It appears to me that you've got basically three "blobs" of high crime in roughly a triangular shape, with the blobs about 5K apart from each other and gaps in between.  Thus it isn't surprising to see positive correlation at that length.
@Glen_b Actually I wasn't sure if I'm using the right term or not. Those probabilities are observed proportions in normative sample so I thought of using them as probabilities for observed values in samples.
I worded the question wrong. the 7 cards do no need to be **drawn** in sequence, rather: after drawing all 7 cards, they need to end up sequential. (that's what a poker straight is. see the example in the question)
Thank you for the additional information
@FrankHarrell: I use a 5% sample of the Swiss census as input, that's more than 300k observations. I have edited the code.
+1 Ease of calculation has certainly played its role in the popularity of Bonferroni. Perhaps more so historically - for instance, it's commonly cited that the need to calculate fractional powers limited the use of the more powerful Šidák correction. By the time that became computationally trivial, the tradition of using Bonferroni had already been well established.
When you say "variables" do you mean "observations"?
Very interesting question and this is by no means an answer, but just curios about motivation - in learning theory we basically aim for obtaining meaningful _upper-bounds_ for this difference for different learning models/methods, why would you try to prove trivial _lower_ bound? :) (probably, not trivial to prove, though)
109 levels presents no problem at all other than computational.  Instead of `polr` use the R `rms` package `orm` function which is made to handle thousands of intercepts efficiently.
@gakera I think you need to have more rep to add more than one link per edit -- I can understand this is not very pleasant when starting on an Q&A website. I was assuming that you would remove your last sentence yourself. On the other hand, I feel you shouldn't expect too much upvotes for providing an answer to your own question, since it is a sort of recap' (useful, though).
I'm not an expert in this area, but is your matrix missing a value in the bottom left corner? Not all rows/columns have the same number of elements.
+1 and welcome to this site! Your link is certainly welcome as it provides useful code and illustrations but your answer is still useful without it (i.e. it's a real answer, not spam or a way to push your website). However, I think a biplot is something slightly different than a scatterplot of scores, I don't see any in your blogpost.
I think if your context is written clear enough, especially in the reference level that you defined, you can just use OR. Readers can understand the odds ratio is for A versus B.
Also I'm not sure the arithmetic mean of $L(\\theta;X)$ makes sense anyway, since $L(\\theta;X)$ is not a random variable. $X$ is observed and $\\theta$ is not random. See: http://stats.stackexchange.com/a/92152/36229.