tttjjj/elt_bench huggingface dataset:
Yes, it is possible to have uniformity without independence. E.g., sample from the unit $n$-cube by generating a uniform grid of $\\epsilon$-cubes covering $\\mathbb{R}^n$ and offsetting its origin according to a uniform distribution on the $\\epsilon$ cube. Retain the centers of those $\\epsilon$-cubes falling within the unit cube. If you like, subsample from them randomly. All points have equal chances of being selected: the distribution is uniform. The result also looks uniform, but since no two points can be within distance $\\epsilon$ of each other, obviously the points are not independent.
This is hardly an exact programming question. There are dozens of good books on regression models and their applications, maybe you should start with one of them? After that, here's a [guideline](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) how to ask a good R question.
If you can use gibbs you can use the slice as well. See the wikipedia link for an example of a slice sampler for the normal(0,1) density. An example where the gibbs is not possible is when you cannot identify the distribution of interest but the density function can be inverted.
The variance is defined as $\\mathrm{Var}(X)=\\mathrm{E}[(X-\\mu)^2]$. And the covariance of $X$ and $X$ is by definition $\\mathrm{Cov}(X,X)=\\mathrm{E}[(X-\\mathrm{E}(X))\\cdot (X-\\mathrm{E}(X))]=\\mathrm{E}[(X-\\mathrm{E}(X))^2]=\\mathrm{E}[(X-\\mu)^2]$ (the same as the variance of $X$). For question 2, look at the [answer of Macro](http://stats.stackexchange.com/a/38722/21054) and just plug in the numbers (the result is $-12$). For question 3, look at the Wikipedia page about the linearity of [expected values](http://en.wikipedia.org/wiki/Expected_value#Linearity). The answer for question 3 is $1$.
Thanks, that makes it clearer. I will recode this variable to ease interpretation.
Ahem. If you observe $n_i$ values on day $i$, do you have the $n_i$ individual values available to you?
Are you sure that it works even for 2 classes? I would expect a heavy overfit.
Use mvrnorm and then exponentiate to get log-normal variables?
And summing the 4 entries gives me 1, 0.1+0.4+0.2+0.3=1, so why is this? Can you tell me the mistake it will be better. :)
I tend to reserve the word correlation for *linear correlation*, and use *dependence* for nonlinear relations that may or may not have linear correlation.
Again, this probably shows my inexperience, but would you be able to provide a definition for contingency tables?  It may also help others who come across this question.
Recursive feature elimination has been proposed in the case of RF (this is quite different from simple screening based on variable importance); see, e.g., the [varSelRF](http://cran.r-project.org/web/packages/varSelRF/index.html) R package and references therein.
If you are ONLY interested in monthly forecasta , I would probably select monthly data. If you are either interested in daily forecasts OR weekly forecasts OR creating updated monthly forecasts as you come through the month then I would strongly recommend daily data analysis.
@scottyaz Also do you realize that JAGs uses precisions (1/variance) rather than variance for the scale parameter of the normal distribution? So the distributions you are sampling from have variance =.01
That's not a bug, it's a feature!  :-).  Seriously, isn't this a desirable behavior?
Dear Max, Thank you very much! Is it OK if I ask another question here? I am finally using elastic net and getting the optimum alpha and lambda after grid search. To get the variables selected I then put this alpha and lambda in glmnet (using the same train and test sets) and predict on test set. I get the same AUC etc as using the prediction function in CARET. 1) is there a way to obtain the selected variables and coef in Caret directly 2)is there a way to extract the selected variables and coef for each fold and iteration during training? many thanks
The edit is on the right track :-).
@Greg: from aplpack package. http://www.oga-lab.net/RGM2/func.php?rd_id=aplpack:faces
I updated the title, is that a better name for what I mean for "clustering algorithm"? I agree being more precise can help searching for future users. Feel free to add a more precise naming if you feel it might help. :)
Of possible interest: [When to use (non)parametric test of homoscedasticity assumption?](http://stats.stackexchange.com/q/2591/930).
There is a difference between the mean and variance not existing and them being infinite.
@whuber Thanks - back to the drawing board on this one. Any ideas is there are any successful examples about creating a correlation structure using the distance between points?
in what year did he write this?
Do you want something like a Sankey diagram (see: http://stats.stackexchange.com/questions/26578/best-way-to-visualize-attrition-using-r)?
@ahfoss: can you explain why this answer is incorrect? If it you are interest, I can post this comment as a new question. Let me know.
Regarding the use of lambda.1se, here's the comment in the LASSO paper, citing ESLII: "We often use the 'one-standard-error' rule when selecting the best model; this acknowledges the fact that the risk curves are estimated with error, so errs on the side of parsimony (Hastie et al. 2009)." See p.18: http://www.jstatsoft.org/v33/i01/paper
+1. I might link also to my own answer mentioning QDA http://stats.stackexchange.com/a/71571/3277.
If you think PCA is worth an entire book, you don't take FA very seriously, and vice versa. I doubt that the books focused on PCA (Jolliffe, Jackson) would be regarded as good treatments of FA by FA specialists or enthusiasts. I don't really know the reverse literature, but the reasons underlying confusion and controversy also imply that impartial, dispassionate treatments will be elusive.
Any further thoughts? I've been reading several things that suggest that large heteroskedasticity-consistent standard errors does imply multicollinearity, and also endogeneity. I'm thinking that endogeneity might be the bigger problem here.
I wish they had a like-button that said "this isn't an answer, but it is worthwhile material".  :)
So how about violin plots with, ideally with mean, sd and outliers drawn?
Actually, I am ready to do that and I applied but it was not correct. Indeed, I do not know what will be the starting point. In case of genotype, I used 0,1 0,1 data. Now in order to calculate similarity between groups what data shall I put? Moreover, if you pls say how to arrange the data for one group,then I can arrange  total seven group's data.Thanks
For a "complex regression", you get R-square, but you don't get r.
Thank you very much for your help. Highly appreciated.
If you give us the full reference of Patterson and Thompson 1971 (including page numbers) then we don't have to look it up.
Even if the incorrectness of the statement didn't bother you, I corrected the Wikipedia entry for the moment.
$J(A,B) = \\frac{ \\mid A \\cap B \\mid}{ \\mid A \\cup B \\mid} - 1$
@Alexis I added a plot to give a better view of whats going on.
Interesting question. You may also be interested in reading [is-normality-testing-essentially-useless?](http://stats.stackexchange.com/questions/2492/)
this question has been asked several times on this list. [This](http://www.stat.ufl.edu/~casella/StatDesign/WebRPrograms/Diet.R) is the short answer. I write a detailed answer later in the day.
"Hours of sunlight and daily temperature are usually correlated" -- I was wondering if anyone would catch that! :)
This is a rather unenticing request for someone to read a whole paper & explain the statistical analysis to you. Why don't you start by explaining what your thoughts are and picking out anything that puzzles you?
Peter,
Thank you for the response!  I am very limited seeing I’ve already collected my data, I now see the complications by multiplying both responses.  I was hoping for a quantitative  way to measure risk in this way.  By not combining the scores and testing each as separate IV’s…are you saying the first question (severity) isn’t ordinal?  But should be treated as nominal?
Thank you so much for your answers! I wish I could mark all as part of the the final answer. :)
This has to be my favorite.
Is this homework?  If so then you should tag it with the self-study tag and tell us what you have tried, what you don't understand, what your course policy is on help/hints on the net, etc.  If it is not homework you should still tell us what you have tried so far and why you have not been able to calculate your own p-value.
@Grega, I added an answer to your comment as an edit in the above answer.
Hello, thanks for your answer. I like your suggestion to use the t-distribution, it seems indeed more appropriate. I don't think it will solve my problem completely (see my comment above).
+1 A quite well-developed answer!
I believe the Encog code is incorrect. The paper specifically discusses H (# of output neurons) and N (# of input neurons) for the given hidden layer in the 2-layer network. The "slicing" is done based on that layers input and number of nodes. It would be illogical to apply information from the input layer to every other layer, however deep.
Is 1 also the minimum value? Are these counts? Likert scale data? Binary data? something else?
+1 for emphasizing the need for both statistical and substantive expertise.
FWIW, if your means are 2 SD's apart, that seems like a pretty big difference to me. It will depend on your field, of course, but that's a difference people would easily notice w/ the naked eye (eg, the mean heights of US men & women age 20-29 differ by about 1.5 SD's.) IMO, if distributions don't overlap at all, you don't really need to do any data analyses; at a minimum, w/ $N$ as small as 6, $p$ will be <.05 if the distributions do not overlap.
In the example you're discussing the power of the test, not its size.  The null, by the way, appears to be $p=0$, for which the distribution is degenerate (an atom at a single point): that's about as far from normality as one can be!
"*does it mean these two distributions are different with a 95% of confidence?*" -- no. If those are actual probability densities, they chance they're different is 100%, since we can see they differ.
(1) (a); (2) The variables or attributes of interest are specific measures, such as ratio of adverse events to patients.
Is there an issue with using ordinal logistic regression?
As far as I see, this is not about pagerank. IMHO, the only overlap is that user paths will most likely correlate to site design (links), but that's it. Aside, the provided link is not working.
I think you'd be better asking this on math.SE, as the question clearly applies equally to anything that can be represented in $n$-dimensional space, not just a probability distribution.
Damien, please read the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) for the self-study tag. What have you tried?
Sorry, I wasn't clear, have tried to clarify.
(+1) The convergence proof is elementary: for $\\omega\\ne 1/2$, let $N$ be any integer greater than $1/|1-2\\omega|$. The existence of such an $N$ is guaranteed by the [Archimedean property](http://en.wikipedia.org/wiki/Archimedean_property) of real numbers. An easy algebraic calculation establishes that $X_n(\\omega)=0$ for $n\\ge N$.
Concerning the known $\\sigma^2$ question, one can imagine cases where this becomes interesting and important. Suppose (say) you are a physicist taking measurements of the speed of light *in vacuo* in order to test a theoretical prediction.  You are employing a measurement apparatus known, through long experience, calibration, and theory, to have a measurement error standard deviation of 0.01 m/sec.  When comparing your measurements to the prediction, should you use the known value of $\\sigma^2$ or the estimated variance from the sample?  A good case can be made for using the known value.
I'd give GPower a chance.  With 20 or 30 minutes exploring it, you'll probably find it's very manageable--at least for procedures like correlation, not necessarily for a complicated regression model.
@MartinO'Leary: I'm not sure how well the question is posed but I was looking ideally for a way to do stratified sampling for both regression and classification.  I just used a simple classification problem to write the question.  I do some work with securities prices where a large portion of the securities move in a small range and a small percentage have large ranges and was curious whether strat.samp. might help.
Are all 94 districts in your data?
@whuber, the biomarker is an explanatory variable. Normal practice in this field is to discretise it as 0-1, 1-10, 10+ or sometimes just 0-1 and 1+ (ie elevated or not elevated). I had the idea to include it as a continuous explanatory variable. Although the dataset has 200 variables, clinical guidance and prior experience suggests to use 10 of these in the final model, hence I was thinking of imputing the values >50 using some of the other variables.
You seem to have answered your own question as to how to *calculate* fitted values; what to *call* them (apart from 'fitted values'), or how to describe them unambiguously in words rather than equations, would be different questions. Conditional geometric means? Geometric conditional means?
Wow, thank you very much @rolando2 for the detailed answer! (also thanks to Jeromy and Thomas!) I'll try to implement some of your suggestions and update this thread with my conclusions.
I want to get a "normal" distribution with the appropriate hump at 0.5.
@ Eduardo: I agree that the dependence is not modelled and that this is somewhat problematic, thanks for pointing it out. I'm not sure how random effects would help -- since the binary outcome `win_home` is constant at the level of grouping (i.e. for all time slices for any given match it's either 0 or 1), including, e.g. a random intercept, for the matches will just result in huge problems with separation in this context.
Charles is on to something.  Unless there are patients staying more than one year, it doesn't matter whether the time period under consideration is 15 years or 1 year:  the mean LOS is still Days/Admissions.  Dividing by 15 will give 1/15 of the mean LOS.
it’s a colname!
What would be the hidden states if he would use an HMM?
I've tried to do some minor cleanup; it looked a bit like this answer was posted in a bit of a rush compared to the thoughtful and careful exposition in most of your posts. Feel free to edit further or roll back. Cheers. :)
Where do those values for start-end values come from? Do you have some justification for using that formulas?
@user877334 yes, exactly, but if you look at the discussion above, you will want to be sure that the model I'm using is the same as the one that you are using!
I think we need to see a reproducible example and the code you tried.
It is not at all clear what your question: *The rho formula divides the covariance by sd of X and SD of Y. Does it result in validy coefficient (or true correlation) for the population.* is asking. I'm having trouble linking this to the question posed by your title: *what is the effect of scaling down covariance in the formula for rho?* (this answer to this is that it will decrease $\\rho$, by the way). This may be a language issue - can you please rephrase/clarify the question?
The algorithm is faster if you sort the categories in decreasing order of probability.  That way, you do fewer tests (on average) per random number generated.
This is a programming question, not a stats question, so we will migrate it to Stack Overflow. FWIW, you have an error in your code `if(i<(m-sd)...`; remember that `i` is an *index*, not the *value* that is being indexed.
For this $f(x)$ to be a pmf, each of $f(0), f(1), f(2), \\ldots$ must be nonnegative (true by inspection here) and the sum $\\sum_{i=0}^\\infty f(i)$ must equal $1$. If you know the series $$e^\\lambda = \\sum_{i=0}^\\infty \\frac{\\lambda^i}{i!},$$ you can apply this to $\\sum_i f(i)$ to show that $f(\\cdot)$ is indeed a pmf (provided that $\\pi \\in [0,1]$) but that it is _not_ a Poisson pmf except when $\\pi = 0$ or $\\pi = 1$.
Approach 2 makes a mistake: it is correct only when $\\mu=\\mathbf{0}$. Even for $D=2$, zero correlation, and unit variances the integral is nasty: it evaluates to integrals of erf applied to trig functions, plus a Bessel function.  If you need an approximation, saddlepoint methods are attractive, especially as $D$ grows.
In R there is the function adonis in the package Vegan that performs permutational multivariate ANOVA. This will generate a statistical test to tell you whether method A is different from method B. This package comes out of plant ecology where you count multiple species (the variables) in different small plots. Related to this is AMOVA, [analysis of molecular variance](http://en.wikipedia.org/wiki/Analysis_of_molecular_variance), where the variables are molecular data. For this you can use the R package ade4, but there is other free and online software you can find at the link.
Actually, the train/test split methodology is subpar, since the performance of your algorithm can vary quite a bit based on your choice of splitting point (I've seen 5% differences...). Better methodology is cross validation.
I assume that the OP wanted to "reduce the conservativeness" of their test to increase power.
The error is likely at your side. Did you read the documentation of hclust? It says about the arguments `d a dissimilarity structure as produced by dist`. Also how do you know the dendrogramm is wrong?
Doc, I believe you do not characterize confidence limits correctly. There's not enough space in a comment to cover all the problems, but one is that a CI is for a particular statistic. If you intend it to be for a mean, as confidence goes to zero it (*usually*) converges to the *estimate* of the mean on which it is based--not a median of anything. You may be confusing confidence with the Bayesian idea of posterior credibility. Note, too, that a CI applies perfectly well to asymmetric distributions--but what matters in its calculation is the *sampling distribution* of the statistic it covers.
Yes, you usually need more than 16 observations, especially with random forest which works by splitting your data. Probably you are using too much trees, when you plot the OOB precision dont you see it go down and back up again after a certain number of trees? model <- randomForest(etc) then plot(model). The caret R package is also quite cute for model building and has some solid theory.
You may be right about using the wrong tools, IDK.  I haven't really found any online materials using the same procedure as I'm thinking of.  I actually am trying to predict a time-series from multiple other time-series (over the same period), but I'd like to predict into the future as well (what good is it to predict a value when it's already available at the time of prediction?). To do the rolling predictions, I'll actually have to predict every time-series though. I guess what I'm trying to do is something like Vector Autoregression, but with ML approaches. Will this approach not work well?
By far your best approach is to use a suitable projection. Using (lon, lat) as if they were Euclidean is tantamount to using the [Equirectangular projection](http://en.wikipedia.org/wiki/Equirectangular_projection), which introduces terrible amounts of distortion.  That's going to be a serious issue in Canada, which gets uncomfortably close to a pole.
After scaling (0-1) dataset from the first slide, I visualize clusters, so I will be grateful to look at the slides and give me a comment. I know that I need to know if clusters make sense from attributes aspects, but I need data miner technical opinion about results. Do you will consider that your job is done if you have an order from your customer, on that specific data, with results as you can see on slides (https://docs.google.com/presentation/d/1A7BbHjfYGiR13NBZSOqDvQjg0fshzPGajWPjEyQCfU0/edit#slide=id.gccc00ff3_20)?
@Elvis Thanks. I tend to think of it as a bit like a two-way main-effects ANOVA model... but for medians rather than means. There's good coverage of it in "*Understanding Robust and Exploratory Data Analysis*", Hoaglin, Mosteller and Tukey (eds); it's also in Mosteller and Tukey's "*Data Analysis and Regression*". Also description and an example [here](http://www.stats.ox.ac.uk/pub/MASS4/VR4stat.pdf)  (starts on page 7).
You infer missing values the same way you'd predict future values, by using the relevant components of the SVD decomposition. Predicting future ratings is exactly the same problem as inferring missing values. If you've got a good way to infer missing values, just use that to predict future ratings. If you don't, then that's what SVD is for.
Just in case it helps for further reading, link to the wikipedia article for KS test: http://en.wikipedia.org/wiki/K-S_Test
can you assume a correlation structure on your data? Your case is a special case of GLMM with logit link, but the correlation structure in the time series data must be modeled correctly to get a reasonable answer.
@pepsimax: I suspect that I still did not understand what your data is like, so I'm not completely sure a) wheter LDA is appropriate, nor b) how you should construct your call. Maybe you could show us a small part of your matrices?
It refers to the probability density function
Thanks gui11. Good general answer.
I listed nice alternatives for performance metrics here http://stats.stackexchange.com/questions/71720/error-metrics-for-cross-validating-poisson-models/71936#71936  Here are some graphical methods listed http://stats.stackexchange.com/questions/70558/diagnostic-plots-for-count-regression/70581#70581Also, instead of RSME
I don't quite understand your question. Are you wanting to try to find the *true* distribution in finite samples of this quantity? Or, are you looking to understand the moment-matching approach in Section 4?
+1 great answer. Also, this might be interesting: http://www.wolframalpha.com/share/clip?f=d41d8cd98f00b204e9800998ecf8427eth176m3og3 http://www.wolframalpha.com/share/clip?f=d41d8cd98f00b204e9800998ecf8427epnvkjtrtd4
If you don't find a solution to the problem maybe you should hand in a bug report to the maintainers R2jags?
It depends on the questions asked in the questionnaire. I came across a clinical study dealing with erectile function and sexual life (wife + husband), in this case the role of the respondents are not symmetric.
Highly voted questions with the recommender-system [tag](http://stats.stackexchange.com/questions/tagged/recommender-system) should give a good start, [this question](http://stats.stackexchange.com/questions/72582/simple-recommender-system-where-to-start) in particular.
@Nesp, correct. Someone else calculated the [arithmetic mean](https://en.wikipedia.org/wiki/Mean_value) $\\bar{\\Omega}(i)$ for each $i = U_\\infty$, then took all $\\Omega(i) > \\bar{\\Omega}(i)$ to calculate the upper error, using the formula for the [standard error](https://en.wikipedia.org/wiki/Standard_error). The same for the points below.
To learn about Bayesian statistics? Yes. I recommend Peter Hoff's First Course in Bayesian Statistical Methods.
Does anybody actually have normal data? How non-normal are we talking? With 110 observations mild non-normality may be of little consequence. If you're constructing prediction intervals it might matter a lot more. What kinds of inference were you performing?
This is theoretically incorrect simply because none of the X & Y variables are dependent variables. Hence, I doubt if doing a regression would be the right thing to do.
Anyone who can help?
it turns out you don't even have to complete the square in this case. Everything you need is already there.
Note that the coefficients attribute (a 2x4 array) of the `summary()` output can be extracted using the `coef()` command. If you just want the estimated standard error for the coefficient of `rprice2`, use e.g. `coef(summary(mod))[2,2]`. If you want the corresponding t-statistic, use `coef(summary(mod))[2,3]`.
I should have specified that I'm *not* talking about kernel methods but about data that has a lot of features.