tttjjj/elt_bench huggingface dataset:
Fair enough, I'll make it an official answer.
Let me back up.  The problem as originally stated can be dealt with using linear splines with intercepts restricted so that the segments meet at the join points (quartiles).  Handouts at http://biostat.mc.vanderbilt.edu/rms show how [in the R rms package you can use the lsp function].  But there is no reason to assume that quartiles are where slope changes should happen, and cubic splines are likely to fit better, without having elbows in the fit.
"If a process generates truly nonrandom patterns for short sequences, it isn't a random process" is, I am afraid, wholly incorrect.  For instance, in any 100 flips of a fair coin, we would *expect* to observe six heads or six tails in a row--and that is a "truly nonrandom pattern for [a] short sequence" by almost anybody's meaning of "random." I suspect you meant to write something that needs to be more carefully qualified, such as applying "all" before "short sequences."
*Normalization* usually means fixing the range of your data to the $[0,\\ 1]$ interval. I think you mean *centering*, which is setting the mean to $0$ (see my answer here: [how-to-verify-a-distribution-is-normalized](http://stats.stackexchange.com/q/70553//70555#70555)). What does "SUR" stand for? What constraints are involved in your situation & where did you hear that they have this effect? Also, why would normalization have to be done? (Cf: [When should you center your data & when should you standardize?](http://stats.stackexchange.com/questions/29781/))
Multiplication of pdfs should give a joint pdf (assuming the random variables in question are independent), not a single pdf. If you meant _convolution_ instead of multiplication, then that gives the pdf of the _sum_ (again assuming independence) and the variance of the sum is the sum of the variances; hence larger than the variance of either. Finally, a likelihood function is _not_ a probability density function, and when you carry out the multiplication you talk about, you are not multiplying two pdfs (well, actually you are but in a sense quite different from the one you seem to mean).
@mpiktas: Thank you very much !!! It works ! I did not realize that the default initial value is zero for all variables in R. On the other hand, I have still to investigate what are the initial values in SAS. Again, Thank you very much !!!
Long names are not much of a problem if you use a decent IDE, which offers tab-completion, so +1 for descriptive names. But don't make the names too long, because that can make code difficult to read.
Many of the optimizations available optimize the choice of initial locations. This is not much use for your application. However when trying to optimize K-means with a profiler, I found by far the biggest improvements were gained by heavily optimizing my distance calculation. I was working on the Earth's surface and was able to unwrap some of the trigonometry.
...or, I would rephrase it: not "uncorrelatedness assumption is made" but rather "covariance structure is not taken into account" when computing multivariate variability. Really, it is reasonable: whatever the covariances, the overall variability is the same, - if you replace a covariance matrix of variables by the covariance matrix of their principal components the trace of these matrices is the same.
Hi Nameless, thank you! Using adjusted scores would be ideal but then the demographics used (age, years of ed) would be controlled for multiple times and have an unduly large effect. By this I mean that I would control for IQ, age and years of ed, when IQ has already been calculated by controlling for age and years of ed. The other predictors (mood, verbal fluency, illness perceptions) don't account for demographics and so only some of my factors would be adjusted, and others not. I think I'll just have to bite the bullet and accept a less accurate estimate of IQ... Lesser of two evils...
100000 probabilities with average .00001 ?? What Kd trees can give you is the nearest few neighbors of all input points or many grid points; if you read Python, see [inverse-distance-weighted-idw-interpolation-with-python](http://stackoverflow.com/questions/3104781/inverse-distance-weighted-idw-interpolation-with-python)
+1 This reflects a thoughtful and informative effort to understand the question and what motivates it.
The number of patients might make a difference in the answers you get. With patient sample size>20 would consider recommending mixed effects models.
I wonder one thing about computer vision methods vs. statistics methods: performance in the first is a must. Maybe there's a trade-off between performance and "correctness", and computer vision and statistics have different weights for those variables.
Have you tried `source("arrFile.R", echo=TRUE)`?
I vaguely remember I have seen somewhere a hypothesis test for H0={Markov} vs H1={Markov order 2}. This could help.
@probabilityislogic I don't follow, can you provide an example?
In addition to the nice `mice` package in R, the `aregImpute` function in the `Hmisc` package may be of interest, using predictive mean matching for missing categorical variables.
... **U.S.** military units were in Europe during the **First World War** (the dates 1917 and 1918 tell the story).
If you're using R and trying to get the same random variables from a distribution, you can set the seed prior to generating the variables. types set.seed(100) (you can use any number, 100 is just for example), and it should generate the same 'random' variables for your distribtion each time. So if you want 100,000 random normal variables with mean=0 and sd=1, just do: set.seed(100)
rnorm(100000,0,1) Is this what you're trying to do?
(+1) Sidenote: Is there a specific reason that you haven't accepted an answer in the majority of your questions ? Why didn't you post a comment when the provided answer did not solve your problem ? E.g.: http://stats.stackexchange.com/questions/9947/centroid-matching-problem
I edited somewhat to comply in notation. Thx.
Andrew Gelman's blog post, ["Adding an error model to a deterministic model"](http://andrewgelman.com/2012/02/adding-an-error-model-to-a-deterministic-model/) is related
+1, that helps. I now see the phrase "at least one of two" and the word "just", which I must have skimmed over.
Assuming "$\\bar{X_n}$" is a typo for the mean $\\bar{X}$, the assertion is false in general: you need more assumptions.  E.g., take $n=2$, $X_1$=$0$, $X_2$=1, let $Y_1$ have any distribution with a positive variance, and suppose $Y_2=Y_1$ (*i.e.*, the responses are not independent). Then the MSE of $\\beta_{ols}$ is $0$ while the MSE of $\\beta_{ls}$ is strictly positive.
@whuber Thanks for your comments. The link is very helpful, and my problem is very similar to the problem in the link. I have to admit that I have vague knowledge about the effect of these three factors when it comes to the problem of selecting the best point pair.
Depends whether you're talking about climate or short term weather. There are lots of predictions flying around about average temperatures over the next couple of decades. It's not going to tell you whether to expect a rainy birthday in 2017.
Any chance for a "before" and "after" visualizations? I'm really not sure (even after going as far as trying your algorithm on some artificial data) what do you mean by "denoising" here?
Hi gd047. Thank you for the link.  The things I know about these models are all linked to from here: http://www.r-statistics.com/2010/04/repeated-measures-anova-with-r-tutorials/   If you'll get to know of other resources - I'd love to know about them.  Cheers, Tal
You have caught a subtlety that often goes unrecognized: "more extreme" needs to be measured in terms of *relative likelihood of the alternative hypothesis* rather than in the obvious (but not generally correct) sense of being further out in the tail of the null sampling distribution. This is explicit in the formulation of the [Neyman-Pearson Lemma](http://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma), which is used to justify many hypothesis tests and to determine their critical regions (and whence their p-values). Thinking this through will help answer your first question.
By ARMA+GARCH you mean ARMA for the mean eqn and GARCH for the variance  eqn?I read in Statistics and Data Analysis for Financial Engineering that but there was something else about ARMA/GARCH model for errors and I didn't know if both were the same thing. thanks
What was the context? There's only one right answer here, but how best to explain it depends on how much you already know. If you can say what the context was, I think I can write a more helpful answer.
Homoskedasticity literally means "equal spread". That is the (population) variance of the response at every data point should be the same. One of the observable ways it might differ from being equal is if it changes with the mean (estimated by fitted); another way is if it changes with some independent variable (though for simple regression there's presumably only one independent variable available in most cases, so the two will be basically the same thing). You could imagine a situation where the mean changes with $x_1$ but the spread changes with $x_2$, which itself is not related to $x_1$.
Because your speculations about normal distributions and standard deviations could send people off in unproductive directions, I have edited the title and tags to better reflect what you are trying to accomplish.
This is going to be a tough one, because Likert scales don't produce the right kind of data. You'd want interval data for your equation, because you'd want to be able to treat responses that are the same scaled distances from their respective minima as equal. Ordinal data are not so nice as this. Consider a related problem, "[Is 0 a valid value in a Likert scale?](http://stats.stackexchange.com/q/32389/32036)" If you had all the original data and not just scores, you might be able to use a partial credit model; otherwise this is probably doomed to be an inexact comparison (more so, that is).
This is cross-posted from http://metaoptimize.com/qa/questions/5296/question-on-mle-and-em, and has been closed because it lacks a clear question. Could you clarify your problem in this particular case?
I can't tell from your question, are your measurements w/i each regime independent?
I am having the same problem. If I pass the custom matrix into the gnls function I get: Error in Initialize.corSymm(corSymm(corCust, fixed = T), data.frame(x = x)) : initial value for "corSymm" parameters of wrong dimension If I just pass through the lower triangle of the matrix I get: initial values for "corSymm" do not define a positive-definite correlation structure Did either user ever find a solution to this? Thanks.
I don't think you are right in the first calculation. If I'm not mistaken, the agreement by chance should be 0.5, giving a kappa of 0.
Noob R user here: Error in data.frame(t = days, values = c(as.vector(data2), as.vector(cum)), : arguments imply differing number of rows: 1063, 1300, 2
+1 The residual plot tells the story in such a way that one can't miss it. Indeed, it shows why the OP's 'looks linear' is often misleading - many curved functions can look 'nearly straight' if we only look at a few points not close to a turning point. *If you think it's linear, take that line out and see what's left over!*
@Iterator Sorry if I gave the impression I was offended. Very much not ;). It's just something I get because I sit square in the intersection between mathematical epi and observational epidemiology, and people in one area are *flabbergasted* the other area exists. I'll edit my answer with a non-SIR type example.
But now I am thinking about our binary target and variables that for the minority class vary but for the majority class are all the same. EX: we are predicting if an employee will continue employment with our company or terminate employment. We have a variable called reason that gives reason why an employee left. For the minority class (i.e "leavers") this varies (i.e. "Better career opportunity", "pay not enough", etc). For the majority class (i.e. "stayers") its all labeled the same thing: "stayed". What about these type of fields?
In my experience, citing Jarque and Bera usually flags that the authors are economists or econometricians. Occasionally it can be useful as flagging to discerning readers that there is lack of awareness in a paper of mainstream statistics literature.
Nice advice @onestop but unfortunately all Mahalanobis distances are quite similar for all observations in question.
FYI, for the sake of interpreting the Gender effect, I would recommend coding one gender 0, and the other 1, which is by far the most common method for coding a categorical variable (at least in the social sciences).
Hi, I'm probably a same level machine learning guy as you, but I want to point out some issues. I don't think there is a "best" model for a job, you simply try different models, different parameter values, and in the end, choose the one that works best for you (and call this a fancy name like parameter selection or cross validation :D ) , I don't know much about NNs, but usually what people do is try different values for the parameters (like hidden layer size, etc.) simply by a for loop, and use the one that results in best classification accuracy.
"*I wonder if I can use chi2 test while there are two unequal groups.*" --- That was already covered in my answer. The answer is yes, though the direct binomial test, while essentially equivalent, is probably more readily understood/informative.
are you trying to perform a network meta-analysis? http://www.bmj.com/content/346/bmj.f2914
what tests are you referring to?
Does `rxDForest` have a non-formula interface (e.g. `rxDForest(x, y)`)? If so, using it may be more efficient. Also, you can probably just use `Class ~ .` as the formula instead of writing out a potentially huge one with each term. I doubt that this is the cause of the error but one never knows.
[Get thee to a database](http://www.enotes.com/shakespeare-quotes/get-thee-nunn-ry), why woulds't thou be a breeder of analytical sins?
I used hierarchical clustering but I think  I did not use the proper distance function because it doesn't regroup curves with the same shape but only those that are close enough in term of euclidian distance. I used method Ward
Thanks! Based on the answers I guess I should start with the little SAS book for now.
@Infinity I added some clarifying remarks.  You don't average out circles; rather, you average out (or "smear") all the probability *across* each circle so that no matter what you started with, it ends up looking like my picture (with circular contours).  If the original distribution was truly circular, this averaging doesn't change it.  Thus, comparing the distribution to its averaged version tells you how far it is from being circular in the first place.
Do you have any good reason TO EXCLUDE THE INTERCEPT? A model without intercept is usually not sensible!  Without a **very** good reason to exclude the intercept, you should include it!
+1. Isn't also called: relative root mean square error (rRMSE)? , cc/ @celenius.
Not the beta values alone, but the predicted values for each case.  If you look into texts on logistic regression you'll find formulae for converting the equation for the logit into an equation for the probability that any given case would be a "1" on the dependent variable.
Ah, yes, thank you very much. I see the resampling 'trick' is not doing anything magical, at the individual level it's really just replacing a continuous value with an a Bernoulli sample. Although this did bring up another question in my mind, in the population Monte Carlo work is the resampling stage really that fundamental?  I mean one could define the next proposal (kernel mixture) using the continuous (normalized) importance weights just as well.  Likewise, in the D-kernel variant, the individual kernel counts could be continuous values right? Not saying this is better, they seem the same.
Thank you for your answer. I've thought about that. As I had a little group of subjects available I took 10 measures from each one. This because I have 23 predictors.
Since $H$ and $S$ are dependent, the assertion that their joint distribution is also normal is an _additional_ assumption. Do you make this additional assumption?
@Mukul For me, this would serve just to understand the concept. In real life, I have never segregated the data and check the variance chunk by chunk (too many tests, and the cutting of the sample can also be controversial.) You can simply plot residual$^2$ against the predicted value to observe if there are any violation of constant variance.
Let me try to be clearer. First, `lm` is ordinary least squares regression: the only randomness it assumes is in the *dependent* variable.  Second, $X^\\prime X$ does *not* estimate second moments: it has to be divided by the data count $n$ for that to occur and even then it is estimating *raw* (non-centered) moments.
I've read that paper and it seems like Stochastic Complexity fixes the problem of not being able to distinguish between models of same dimensions, but introduces a problem of sometimes not being able to distinguish between models of different dimensions.  Geometric distribution is assigned infinite complexity, surely not what we'd expect for such a simple model!
+1 for a couple of the other answers but I think not emphasis put on what I think is the OP's misreading of "explaining away".
When and how much the dealer will buy will depend on inventory as well as expected sales, so you need data for that as well.
Yes, it has a statistical basis: and that basis shows your technique is not a valid way to identify outliers! If you use *Mathematica's* `NonlinearModelFit` procedure, it will give you influence measures, prediction confidence intervals, prediction standard errors, and standardized and studentized residuals: all of these are useful for identifying outliers (of various types).
Can you post `dput(lesna[1,])`? That will have all the the factor structure needed to make a correct argument to `newdata`.
+1 For pointing out the collinearities.
What's the objection to line plots?
Another thing to consider is using Ridge Regression, which was specifically designed to deal with multicollinearity. Or perhaps some other penalized regression approach (lasso, Enet, Mnet, SCAD, etc.)
It does not mean that you need to store it in that way. If you have that many documents and terms, then you can store them in a _sparse_ way. For example, even though each document has 300 000 columns, the 99% of them are 0s. So instead of storing full vector (e.g.. `[0,1,0,0,0,1,0]`)
you can store just `[1,5]` which means that at the indices and 1 and 5 there are 1s and the rest are zeros.

I'm pretty sure R can use those sparse matrices, you can check how to do that in Java, there must be a way :)
"Engage", or distract? Colour can be ok, but you're ultimately here about data, and the aesthetics should serve the data, and not the other way around.
Welcome to the site, @nub. We hope to build a permanent repository of statistical information, as such, we worry about the possibility of linkrot. Would you mind giving a summary of the info in that thesis in case the link goes dead?
If this is homework, please read the FAQ and update your question accordingly.
I've edited my post to add some example vectors from two datasets that I use.
ok, thank you! and i actually never thought about why I couldnt say we accept the null hypothesis. thx for making that clearer aswell!
Yes, this is true. Often times however, Gibbs-sampling and the Metropolis are being used in conjunction. So conditioning on some variables might give you a closed-form posterior, while for others this is not possible and you have to use a "Metropolis-step". In this case, you have to decide for which type of Metropolis-sampler (independence, random-walk) you go for and what kind proposal densities you use. But I guess this goes a bit too far and you should rather read into this stuff for yourself first.
I guess that is exactly the course the original poster mentioned...
Well, basically this site is about data analysis and statistical modeling, as described in our [FAQ](http://stats.stackexchange.com/faq), and not about software issues.
No, this approach is not valid, because you have only 10 independent observations, not 400, and there's nothing you can conclude from comparing just two numbers, either. However, you're not asking a question that will help you much. Instead of asking about one invalid form of analysis, why don't you share your full dataset (all 10 numbers won't be hard to post :-) and ask for advice about how to analyze them?
Have you considered the modified newton's method? It was a method built specifically (as far as I know) to deal with running into singular matrix. You simply make an $n$ by $n$ identidy matrix which you subtract from your Hessian to avoid a matrix that does not invert.  As for your second question, Newton's method can run into problems that it finds local and not global maximum, so you should try different starting points and see what you get.  It can also start moving in one direction (the wrong one) and simply never converge, so starting point selection is very important.
I haven't read the Stanford article. However, I don't find the reasoning in the quote compelling. It may well be that in some population, treatment B is better than treatment A. This doesn't matter. If that is true of some population, it is only because the population's characteristics are confounded. You are faced w/ a patient (not a population), & that patient is more likely to improve under treatment A w/o regard for whether that patient has large or small kidney stones. You should choose treatment A.
Have you consider Unit Testing with RUnit to verify theoretical behavior?
Sorry. "Error in chol.default(W): the leading minor of order 4 is not positive definite." I have double checked, no monitors on constant nodes.
What do you need explained? I think that, if you don't understand the output with the explanation I gave, you may need to learn more about regression.
Interesting answer. I just completed the course and it was outstanding. They are starting it again at http://jan2012.ml-class.org/ ...
I finally managed to do this and I get the following plot
@Graeme Walsh, NSFW, you got it.
You want the probability of $a$ positive from $A$, $b$ positive from $B$ and $c$ positive from $C$ where $a+b+c \\le k$.  So you use the binomial distribution three times and do a lot of adding up.
"*Levene’s test – nothing was significant – 0-hypothesis accepted – equal variances*" -- 'equal variances' is not an appropriate conclusion after applying a hypothesis test. The population variances are almost certainly *not equal*.
it's well known that IRWLS doesn't always converges for the [logit](http://www2.sas.com/proceedings/forum2008/360-2008.pdf). The most comon cause in applications is the issue of [(almost) perfect separation](http://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what/45831#45831) and it is a statistical issue (so this forum is the correct place to ask). Do your non convergences problems happen in situations where the data is perfectly separated?
I asked a similar question at http://stats.stackexchange.com/questions/72570/simulating-from-posterior-predictive-over-many-periods but it looks like yours has received more up votes so far.
I would go ahead and try it before worrying.
No, this doesn't hold in general or for the specific case of betas. There are families that are closed under iid sums (Binomial, Poisson, Gamma, Normal) and there are some that are closed under arbitrary linear combinations. These latter ones are called [stable distributions](http://en.wikipedia.org/wiki/Stable_distribution) and have been fully characterized. The Gamma distribution is an interesting case: It is closed under sums, but is not stable, yet it is *also* closed under means!
Ooops...looks like I wasn't paying attention while reading the paper. I will look at the tutorial, thanks.