tttjjj/elt_bench huggingface dataset:
In particle physics they use 0.0000003.
You should keep in mind that the within estimators requires a strict exogeneity assumption which is necessarily violated if you include lagged outcomes as regressors, e.g. www.nber.org/WNE/lect_2_linpanel.pdf‎
You _must_ set any one category to be reference. It is mathematical, not substantive, prerequisite.
it might be useful if you could edit the textbook names into your question (for future reference) too. Thanks!
Ok, I didn't look in details into this. For k-means, you need to pay attention to the fact that we generally need two outer loops for validating cluster solution (one where you vary the # of clusters and another for varying the seed -- the objective being to minimize the RSS); then you can use the Gap statistic to choose the optimal # of clusters.
I think it's $p_1 = 0.18, \\; p_2 = 0.01$ and $q_1 = 0.32, \\; q_2 = 0.04$.
@Howie the bulk of a single sentence may be tagged `other`, but do all of the sentences in your training set include a named entity? If not, I'd recommend creating a binary flag associated with each sentence in your training data to indicate whether or not it includes a names entity and apply the over/under sampling to whole sentences to amplify the proportion of sentences containing named entities in your training data.
Thanks for advice and refs. So following seem to be the attributes to be taken into consideration while choosing decision trees - 1. Training time  2. Readability 3. Memory requirement 4. Attribute type mis-match  5. Supported class size
Some useful typographical information--but not dispositive, unfortunately--appears at http://en.vionto.com/show/me/Overline.  This source explicitly equates the term "overline" with the $\\TeX$ "\\bar" construct, which produces $\\bar{x}$. However, in Unicode, multiple successive overlines connect up whereas multiple "macrons" remain separate, suggesting that the right way to create an *overline* for statistical typesetting is to use a Unicode *macron,* even though most sources would *call* it an overline!
Looks interesting, thanks.  I have similar problem to the OP trying penalized + coxph on a big data set, wonder if this helps.
What does "zero is not anticipating" mean?
You already get $p<0.05$/$p>0.05$ based on whether the confidence interval does not/does include 0. Any more detail is not possible since the distribution of the parameter from the bootstrap is not parametric (and thus you can not get a probability that the value is 0).
@Harokitty, yes. Keeping in mind that ${\\rm cov}(X,X) = {\\rm var}(X)$, you can apply this property to find that ${\\rm cov}(X,aA+bX) = a \\ {\\rm cov}(X,A) + b \\ {\\rm var}(X)$.
@pixel For SVM certainly (it is called SVM regression), for RNB there might not be an implementation, but it can be done as in RF. If not, and with LERS you may just partition the decision into a discrete groups -- it is a common trick and will be sufficient for feature selection.
Although for my specific problem, that doesn't seem to be it, that's a very good point.
The values of the smoothing parameters are frequently chosen by an iterative process to minimize the in-sample sum-of-squared prediction errors, sometimes penalized when the parameters are near 0 or 1. I don't know if $\\gamma$ refers to the trend, level, or seasonality parameters (there's no standard notation AFAIK), so your other question is not currently answerable. If you have seasonality, you might explain if it is additive or multiplicative in your model.
collinearity won't effect the prediction of the model, only the estimates of the coefficients for the variables involved. For a binary classification issue, logistic models are the gold-standard. And a good base to build on. You can then expand/change you approach as your understanding of the data improves.
By Kendall's tau being a vector means that I converted the Kendall's matrix into vector using P2p() function of 'copula' package so that I could pass the vector generated as an argument to any copula computing function. I computed student's T copula from tCopula() function which takes in vector as an argument for computing multi-variate copula. I wish to do something of that sort in gumbelCopula.I want a copula object of multi variate gumbel copula which takes in either correlation matrix or non-parametric correlations like kendall's tau as parameters.
If there are $0$s and $1$s in your sample, then there is evidence against a Beta distribution because this is a continuous distribution. Do you know how are they generated? Are they observed due to rounding for example?
Does this help? http://homepages.inf.ed.ac.uk/s0562315/progs/index.html#hpylm
Tom Mitchell's book "Machine Learning" is also very good; the style is a bit old-fasioned, but the content is excellent.
@ArthurB. It could also be likely to be a fluke because it would be easy for small imbalances of confounds between the two groups to be responsible for small effects. However without knowing all the important possible baseline differences between groups that could be important we have no way of knowing for sure.
It is the variance which is an unbiased estimator. S is not an unbiased estimate of the population standard deviation.
You mean "prediction interval" not "confidence interval". The latter is for a parameter.
Not exactly...$\\Lambda_1$ is in $\\mathbb{R}^N$ while $\\Lambda_2$ is in $\\mathbb{R}$...which is one of the nice features of a sufficient statistic...dimension reduction. The two are related in that if we let $R_T$ be the rejection region for T, then the rejection region for $\\Lambda_1 = \\{x_i: \\sum x_i \\in R_T\\}$ I hope this clarifies it...the two problems are in different dimensions (literally), but are equivalent in the sense I just pointed out.
"linear regression will often have a good idea of value of f(x) for some unseen x from just the value of the x" not sure what you mean by this either... you still need the parameters of the linear model, just as you would need parameters for KNN (though its parameters are more involved)
Thank you for your answer. However, I'm not trying to understand everything about everything from one book, but reading say, 50 pages about regression would definitely help me a lot for getting at least some reasonable understanding of this subject...
@chl, I asked the question on meta concerning adding r tags, since I felt too, that adding r tag just for syntax highlighting is not appropriate. Google prettify did not work for me.
I'd write to the authors. There are email addresses. You're new here, but please try a little role reversal: People who answer questions do so for various reasons, including the question being interesting and their wanting to help others. But your question is still: Please read through a densely written 11 page paper and tell me what the authors are doing. Would you do that for someone else? Questions work well here when they can be understood by reading what the OP writes, in at most a few minutes. Your question doesn't qualify in my view.
`mim` is not an option for me as I do not have Stata available.
@Nick Cox: Another point of confusion for me is the difference between doing PCA on a correlation vs covariance matrix - what I am reading suggests that doing it on a correlation matrix means you don't need to scale the variables prior to running the PCA (i.e. they're already scaled), whereas you do need to scale with the covariance matrix. Do you know if this is this correct?
+1 but this would be easier to read if you used LaTeX
(1) EMD is earth mover distance (the total work required to move all A's mass onto B, under the optimal scheme for doing so) 

(2) in each sample the weight are for the same 10 people each day.

But the people in each sample are different which mean we have 20 people in total.

(3) at random (4) Hausdorff distance between the 3d point from the first dataset and the 3d point of the second dataset
It's generally a good idea to show a scatterplot of the data before interpreting a correlation coefficient.
possible duplicate of [What is the difference between a population and a sample?](http://stats.stackexchange.com/questions/269/what-is-the-difference-between-a-population-and-a-sample)
The second question is answered, *inter alia*, at http://stats.stackexchange.com/questions/2591.
@jrand Yes, that's exactly what I wrote (although your first equality doesn't accomplish anything: it's just a more laborious way to write the summation).  The whole point--and the thing worth remembering--is that when $\\varepsilon$ is a random variable with a variance and $\\lambda$ is constant, $Var(\\lambda \\varepsilon)$ = $\\lambda^2 Var(\\varepsilon)$.
Hmm is there any reason to use `mlogit` package (which is, I believe, designed with the aim of dealing with complex problems, including using variables of different type -  alternative specific variables and individual specific variables) instead of, for example, `nnet` package with its `multinom()` function? Or instead of any others, which are able to build multinominal logit model?
I'm a little unclear on what you're asking.  Are you asking whether it's a problem if the practice effect of B on A is of a different magnitude than the practice effect of A on B?
Hi. "PLS is the leatherman of ...", "Bootstrap is the leatherman of ...", "Random forest is the leatherman of ..." <- I advise you to be suspicious about such claims. It was just a comment :)
@ilhan, in my reply, I used `N` for both N and n. `N` is a size of a totality at hand, either population or sample. To compute _population_ variance, you _must_ have population at your disposal. If you have only sample you can either compute this sample's variance or compute population _estimate_ variance. No other way round.
This is fairly opinion-ish. Should this be community wiki?
I think the OP is talking about t-scores (mean = 50; sd = 10). I know this must seem a strange terminology given the meaning of t in statistics.
I also just thought of using the `kmeans` function in R.  I really shouldn't ask questions between midnight and 4am.
Hmm, thanks to the "related" questions I found this possible solution: http://stats.stackexchange.com/questions/33433/linear-regression-prediction-interval
(Sorry I could not include graphs,  I don't have permission to share the data.) So there can be a Kalman filter settling time. That makes total sense, looking at the Big Bash (moving seasonality) effect: the first year has movement in the "baseline": in the one case it wanders a bit, and in the second case it has that horrible spike. After that, the baseline (Big Bash indicator 0) runs nearly flat around 0.95, while the spikes (Big Bash indicator 1) go up to 1.10-1.25 (multiplicative effect), which is what I'd expect. The 0.95 baseline I think due to me not centering the 0/1 Big Bash series.
`lm` rather than `ln`
Are you able to let me know how I might use this package to deseasonalise the data? I haven't been able to get hold of that book, however from the package documentation it looks as though it might suit my purposes.
No, it's not what I said. You didn't make it clear that X depends on the history of Z and X. That's a pretty serious omission! How can we give reasonable answers if you leave out such vital information? It just induces us to waste time giving useless answers.
If this question is about "How can I learn R in the next 5 days because my results are due in 6 days?", then it will be closed as off-topic (see our [FAQ](http://stats.stackexchange.com/faq)). If you wonder about learning R when coming from SPSS, you can have a look at the [Quick-R](http://www.statmethods.net/) website. Otherwise, please indicate what you are expecting from responses in this thread: "What are your thoughts?" is too vague.
Thanks vinnief! The panel is already unbalanced - so this would not cause major problems I guess. But I feel my major problem is not clear yet: There are adults and children responses in every time period (though there may be 20 adults and 10 children on day 1, and 40 adults and 4 children on day 2,etc)..how would the dummy look like? The independent variables (like ice-cream cost or ice-cream size) are the mean of the responses. I could take all responses (one dataset) or separate by adults and children (two datasets)..but would an adult dummy in a panel data set work?Thank you all very much!
@Glen_b quite correct. Fixed it
ANCOVA is a special case of regression for comparing categorical variables. Just use regression.
I don't think it's technically valid, but it (i.e. assuming piecewise-constant intensity between events) seems to be a reasonable approximation for a slowly varying process ...
For a one tailed test, you just count how far in it is from the end. For two tailed it's a little more complicated, but still quite simple.
@AdamO, if the $\\epsilon_i$ have a logistic distribution, then this describes logistic regression.
Essentially yes, cross-validation only estimates the expected performance of a model building process, not the model itself.  If the feature set varies greatly from one fold of the cross-valdidation to another, it is an indication that the feature selection is unstable and probably not very meaningful.  It is often best to use regularisation (e.g. ridge regression) rather than feature selection, especially if the latter is unstable.
(i) *stepwise regression* carries dangers; search on it to find numerous posts discussing them; $\\ $ (ii) Even without the stepwise issue, models with different numbers of parameters are not directly comparable on the basis of $R^2$... indeed with nested models and continuous r.v.s, the larger model will always have larger $R^2$; $\\ $ (iii) but even if you replaced that by something that is more comparable, there's still the danger of trying to compare models on the basis of regression statistics alone -- see [Anscombe's Quartet](http://en.wikipedia.org/wiki/Anscombe%27s_quartet), for example.
Although I know there's a logical explanation, I'm struggling to get an intuitive feel for why the beta for `predictor2` switches from 2 to -3.467 dependent on whether `predictor1` is in model and dependent on sample size of `predictor2`. Just to clarify: what exactly does 'realization' mean? When you say 'x 1 SD below the mean' what mean are you refering to?
There are restrictions on the correlations that are achievable. The restrictions depend on the correlations and on the actual distributions. If you have additional requirements, such as that distributions A, B, and C all be in the same location-scale family, that will considerably simplify matters, so please provide any such details you can. It is possible your question is already answered here, including the `R` code: see http://stats.stackexchange.com/q/24257 for instance.
Are you stuck with a mixed effects model? In order to select the best predictive variables and account for collinearity, why not run a penalized regression (eg, lasso, ridge regression, or elastic net) on your model, with each state as a separate dummy variable.
Related question: [Standard formula for quick calculation of scores](http://stats.stackexchange.com/q/20439/930).
Hrm. To my mind, climatology isn't interested directly in measurements, just in patterns. Monthly means and variances are more meaningful than daily measurements. Given that context, how can one look at conditional probabilities?
@Frank **log**-likelihood certainly can't. But *if* (as I mention in my answer) we're careful not to lose any constants (or slightly more generally - as I mention in comments - we are careful about keeping the ratios constant), then in fact the P(data|model_i) values can be computed.
This is going to depend on the nature of the underlying relationships amongst these variables. BTW, is this homework? If so, please edit to add the homework tag.
This display is tough on people who are red-green colour blind. A green dot is good and a red dot is poor. Poor design.
See tables in [here](http://books.google.ca/books?id=gW51UqDZfB8C&pg=PA512&dq=tails+off+pacf&hl=en&sa=X&ei=EWNuU8qiKseGyAT1-YK4DQ&ved=0CC0Q6AEwAA#v=onepage&q=tails%20off%20pacf&f=false).
Great answer! Should the line for calculating `y1` be `y1 = 5 + .5*x1 + rnorm(20)`?
@user603 No, you don't read me correctly. "Outlier" doesn't mean anything - especially when flagged by an automatic procedure in statistical software. There are just as many examples of the important findings of a study being in the "outliers". Whenever you have data you're deleting, it should be for a reason. "They're inconvenient" isn't a reason.
As an example I first thought of old numbers games where they used some decimal point of specific stock reports in the news paper. Of course you would worry about the true randomness of this metric though, so I imagine there are better examples.
e.g. http://hmmer.janelia.org/ This is the one I have experience with and it is used for bioinformatical research. There are many, many HMM implementations out there (C#, Java, python, C++, ...). Did you already search google for it and did not find something that worked for you (please specify why) or did you just not look yourself yet ;) -> sorry I just saw this post is very old, I don't know why it showed up at the top!
I appreciate the kind words, Giiovanna.
I can then add up the number representation of condition present in an area to give me a figure on the quality of health. What I would like to know is if there is any analytical function in spss or some other software which gives me  something near this
Works for me on win 7 x 64. Maybe you can provide the output of `sessionInfo()` ?
See also this existing question on recommended books on [experimental design](http://stats.stackexchange.com/questions/1815/recommended-books-on-experiment-design?rq=1); you may wish to indicate whether this existing question meets your needs or whether there is something particular related to bakery/industry applications that leads you to want a separate question.
I am actually trying to figure out how to measure similarity. I added some info to the question. i have 30 x 4 data which is 120 real world sequences and 120 observed sequences.
(I'm marking this as a community wiki)
@ifreak, to paralellize I would look at just using the parallel package and the `parLapply` or `parSapply` function.
In general, I think relaxing a bit your view of language would help understanding (e.g. accepting that there isn't necessarily one true universal definition for each word, realizing that many things are written quickly without pondering every nuance of every sentence for hours and that people occasionally use words, even technical ones, in slightly different ways). In short, don't read too much into every text you come across.
Thank you Steve. My dataset B already is multiple imputed (not mentioned in the question). As per your suggestion, I will have multiple datasets. For dataset A, I took the annual weight as the mean weight for each quarter (the quarterly survey are independent of each other)
(+1) this night I thought carefully about your suggestions, and decided that time domain indeed is the last thing (due to lag truncation and weak properties in small samples) to try searching for cycling behavior. What I am personally concerned about are the assumptions for F statistics and the small sample size properties of the suggested scheme. Well and probably it is good to start a separate question regarding the optimal window selection, because there are indeed many.
@PaulManta The book chapter Efficient Backpropagation from Yann LeCun is a good starting point for neural networks optimization algorithms. It is freely available from his website.
(+1) For most readers, the very first line under your graph should be more than enough. I was just about to mention the oddity of the iid assumption in the question statement; I'm glad to see you made an edit to point this out! :-)
>Isn't this pretty much the binomial distribution? Well, to parody a quote from an ex-President, it pretty much depends on what the meaning of pretty much is.
$\\widehat{r^{2}_{\\text{true}}} \\sim \\text{Hoyt}$, and $||\\bar{x}-\\mu||^{2} \\sim \\mathcal{N}(0, \\frac{1}{N}\\Sigma)$? Then the characteristic function of $\\widehat{r^{2}}$ is the product of the two characteristic functions as explained [here](http://en.wikipedia.org/wiki/Characteristic_function_%28probability_theory%29#Basic_manipulations_of_distributions). That indeed answers my question. Do you know how we might suitably transform $\\widehat{r^{2}}$ such that its distribution is known without access to $\\Sigma$? Like the Mahalanobis distance, or the univariate $t$ statistic?
@Ladislav Nado. Thank you for the comment! I agree with your suggestions regarding the reporting of variance. In this specific case, do you also find the SE to be too large, or is it statistically/mathematically possible for this SE to occur in relation to the range and number of observations? Any insight into whether or not I have understood this correctly would be appreciated!
For your time complexity point, it depends on how the values are stored. If the values are already sorted, then it is certainly possible to calculate median in O(1) worst case time complexity.
As stated, this question is quite vague. Questions that are self-contained, concrete and well-motivated tend to receive the most attention and best answers here. (For example, if you have a particular problem you are trying to solve, consider providing enough detail that readers can understand what you're trying to do.)
I find this a helpful exercise.  As an argument, though, it is unconvincing because it relies on an unstated assumption: namely, that Mr. Hall will even offer an opportunity to switch and, if he does, that his choice is independent of yours.  For example, if Mr. Hall happened to learn that you intended to switch (and he wished to minimize his losses), he might elect to open a door only if switching would cause you to lose!  In this case, your chance of losing becomes 100%.
(+1) Ah... we cited the same book :)
This is similar to http://stats.stackexchange.com/questions/5700/finding-the-change-point-in-data-from-a-piecewise-linear-function
http://stats.stackexchange.com/questions/51718/assessing-approximate-distribution-of-data-based-on-histogram/51753#51753 already answers half of the question, namely why histograms are best avoided, no matter what you replace them with.
Thanks, for the permutation approach I take samples from the entire dataset at random, without consideration of event / no-event (ie it would be wrong to do the permutation test just on no-event data)?
Spring force is proportional to the deformation, so this is not a least squares regression!
Also see gung's comments on your other post.
Louvain modularity algorithm is fast and steady， i wonder if there is a map reduce version of it.
Ha no worries. The (.)s represent zeros. Since you went with Lasso, you've specified that you want a "sparse" solution (i.e., lots of zeros). If you want them all to have values, set alpha = 0. Now you've went from Lasso to Ridge regression. p-values for glmnet are conceptually tricky. If you google search "p-values for lasso", for instance, you'll see a lot of recent research and debate. I even read one account (source amnesia) where the author argued that p-values do not make sense for biased regressions such as lasso and ridge regression.
Ahhhh ok...I will edit the question now. It seems it is confusing!
@Corone - If there's general scientific agreement that the AUC is a poor indicator, it's news to me, and in health care research I see Concordance or the C-statistic used very frequently.  Can you back up your statement for me?
"Weighting by the denominator itself may be reasonable" - Please could you explain how to weight by denominator properly? I understand  your method of combining percentages but I don't think I can apply it in my data as the "experiments"/or courses are different. Eg if trainer at Catterick won 6/48 = 13% races and same trainer won 1/1 = 100% at Exeter how should I adjust the percentage or introduce new variables for NN/Decision Tree/SVM predictors for a predictor named eg "% success at todays course"?