tttjjj/elt_bench huggingface dataset:
@Stef, your point that any single imputation is inherently limited because it doesn't recognize the difference between the real gathered data and the guess imputed values is absolutely right. Still, single imputation with either noise or random selection is not so bad. In many cases (e.g. in exploratory rather than inferential alalysis) to use multiple imputation is cumbersome and hot-deck or EM remain nice alternatives.
Thank you Peter. Will use a descriptive title in case I have another question.
+1 for recommending `zoo`. It would make your task much easier. `xts` is meant more for package implementors, though I have used it directly.
If you only have betas, you are lacking critical information about the actual variation of the independent (and dependent) variables. Thus the studies cannot validly be compared; a meta-analysis does not seem possible.
@Andre "HW" = "homework."  (I added [tag:self-study] to clarify this.)  FAS: it is important that $\\sigma$ be known.  That reduces your problem to showing that the distribution of $T-\\bar{X}$ does not depend on $\\theta$.  Although this could be done (painfully) using the method you describe, a simple demonstration is possible based on observing that your family of distributions is a *location family.*
Many of the same considerations are pertinent in my past attempt to measure reputation effects, see http://stats.stackexchange.com/a/14352/1036, the critique by whuber and the associated blog post. Although off-the-cuff I would say it is a substantially easier question to address. Such things as questions views and subject popularity are hypothetically not pertinent (or at least should not be as strongly related as they are to voting behavior).
-----  Data -----
Consider a uniform density on $(0,0.1)$. What's the height of the density in that range? Density is not probability.
Apart from a full formal proof, I think I have figured it out that it does indeed converge almost surely to 0.
@whuber - your premise "...if you are reasoning consistently..." has a consequence from the good old Cox's theorem.  It says that if you are reasoning consistently, then your solution *must* be mathematically equivalent to a Bayesian one.  So, given this premise, a CI will necessarily be equivalent to a credible interval, and its interpretation as a probability is a valid one.  And in Bayes, it is not the parameter which has a distribution, it is the uncertainty about that parameter which has a distribution.
There's lots of information about this over at stackoverflow; start with this question http://stackoverflow.com/q/7938883, for example.
Also I think my last statement sufficiently summarizes what I am interested in, and I don't see it as being entirely consumed by formal inference.
A re-look suggests that the answer is good and the comment suggests correctly the effect on mean. However, in the case of variance also, being a random effect, it should not affect the variance. In general, the answer is accepted.
I am still not getting this. How did you get N=3 and why not 4 or 2? Also, how did you divide them into 3 bins with those particular data points. I would highly appreciate any references as I also wanted to see how equal probabilities are done (in terms of chi-square ).
$Y=Z-X$ which should give you something to analyse
static, that link doesn't display any boxplots at all! Do you perhaps have a different understanding of [what a boxplot is](http://en.wikipedia.org/wiki/Box_plot) than others do?
And here's [an intro to mixed models in SPSS (PDF)](http://www.spss.ch/upload/1126184451_Linear%20Mixed%20Effects%20Modeling%20in%20SPSS.pdf).
Also English can be a confusing language with plurals, the plural of dog is dogs but the plural of noise is noise not noises.
The alternative would be to replace the hold-out set by an *outer*, *independent* cross validation (so-called nested or double cross validation), not to skip the validation of the optimized model.
2 parts - 1  the smoothing parameter and finding the quantile part is near synonymous with [quantile regression](http://en.wikipedia.org/wiki/Quantile_regression). The second is allowing the function to be non-linear in particular ways. Popular techniques are smoothing splines (see [GAM](http://en.wikipedia.org/wiki/Generalized_additive_model) for some related background) and both are implemented in the [quantreg R package](http://cran.r-project.org/web/packages/quantreg/quantreg.pdf) (see the `rqss` function). Also related see the [VGAM package](https://www.stat.auckland.ac.nz/~yee/VGAM/).
What do you think would cause the change? That should be the determining factor rather than looking at a likelihood function.
If this were just least squares regression and you wanted a peak at x=10 of height 5, you might write y=-5*(x-10)^2. Multiply that out and you'll see constant, linear, and squared terms. Same idea here for logistic regression. You want to give the estimation process flexibility to choose the location and height of the maximum, as well as the rate of drop-off as you move away.
Questions about programming are better suited for stack overflow.
I don't understand, from the description, your dataset. Can you post a small sample of it?
@ChrisC Thanks. It is become more clear, but I still don't really grasp what this assumption means. The whole point of knowing about the assumption behind the method is to know when you are likely to be violating it. So it would help to list some scenarios where the assumption is not true. When would a lower P value not be associated with a higher probability of the null hypothesis being false?
What resources have you found? Why don't you list them? How did you decide on using a Neural Network? I think your question's too vague and broad.
I think we will need more information. Do you have measures at baseline (before implants)? What do you mean that the ordinal data are normally distributed? What do you mean that "most of the data are normally distributed"? Can you post a sample of your data, or a plot of them?
This question is too local. It is basically loses value everytime a new library for R or Python is released.
@Berkan I don't think either issue you mention (more no event sequences than event sequences and just noise for no event) should rule out the 2 HMM approach. If you took the prior $P(HMM1)$ into account (I've update my original answer in this regard) then you may need to adjust for the unbalanced class distribution (more no events than events), but there are lots of ways to deal with this. See [this](http://stats.stackexchange.com/questions/64163/how-to-deal-with-low-frequency-examples-in-classification/64165#64165) answer I gave for example.
You seem to be using "confidence" in your comment in the sense of a Bayesian credibility rather than as defined in the statistical literature. As an extreme example of the distinction, a procedure that uses a spinner to return $\\mathbb{R}$ with 95% probability and the empty set with 5% probability is a valid 95% confidence interval (albeit a terrible one, obviously, because it entirely ignores the data!). That shows why there is more to evaluating the quality of a CI procedure than checking its coverage, but it does not demonstrate any inherent flaw with CIs.
@ttnphns: In your linked example the number of points is much higher than the number of dimensions. If the situation is opposite (like in this question), LDA will hopelessly overfit. Reducing the dimensionality with PCA before doing LDA will often act as a sort of regularization, preventing overfitting and dramatically increasing the LDA performance.
are you asking about how to specify the knots in $R$ (i.e., via arguments to **ns**) or are you asking about strategies for deciding on where to place the knots?
(i) You'll have to give more context about exactly what you did. (ii) the distribution of the actual [ratio of range to sd](http://en.wikipedia.org/wiki/Studentized_range) changes with sample size. [relevant post](http://stats.stackexchange.com/questions/69575/relationship-between-the-range-and-the-standard-deviation)
I tracked down the contributor who posted that pseudocode; it looks like he's still around, so I left him a message with a link to this question. So maybe he'll show up here.
Your first remark isn't complicated Michael, but it is a bit vague.  It would be terribly tedious, and highly unlikely of this questioner (given the wording of the question), to have tested for normality of y at each b.  Given that it wasn't mentioned that the residuals were what was tested for normality, and it's kind of implied normality was assessed before the model was calculated, it's reasonable to advise the residuals be examined.  A questioner as naive as this one might actually take your comment as evidence that they can just check normality of the y values.
Shouldn't we merge this question with [the same one](http://stats.stackexchange.com/questions/26/what-is-a-standard-deviation) asked last year?
@IrishStat You ask the right questions - maybe you want to post an answer containing questions and answers. My approach was: he/she wants to do a forecast. Instead of writing 100 lines here I give him/her a link to the best book (online, free) I know with all questions and answers and R code. I prefer this.
So long as you have an IID sample, this is always the variance of $\\bar{X}$ (assuming the variance of each draw is $\\sigma^2$).
Hi Adam. Is $U$ finite? Also, how do you get $|\\mathrm{Coll}| = 2^{2^n}$? Suppose for example $U = \\{1,2,3\\}$, $A = \\{1\\}$ and $B = \\{1,2\\}$. So $n = 2$ and $2^{2^n} = 16$. However, there are only 8 possible subsets that can be generated by finite intersections unions and complements of $\\{U,A,B\\}$. You'll also need to specify some form of probability model to get an answer here. That piece is still missing. Cheers.
+1 I like this because (a) a graphic usually reveals much more than a test and (b) it is practical.  I'm a little concerned that it has an arbitrary aspect: its appearance will depend (strongly, perhaps) on the order in which the data are given.  When the "partial variance" is due to one or two extreme values, and they come near the beginning, this graphic might be deceptive.  I wonder whether there's a good solution to this problem.
I'm not sure I understand why missing data means your models are not nested. Is it because cases are getting deleted if they don't have values for predictors in one model but not getting deleted in the more minimal models? That would pose a problem for AIC/BIC too, because they only provide meaningful comparisons when used on data sets with the exact same set of observations/cases.
Since 'normalize' is used to mean several different things in contexts like this, precisely what do you mean by "normalize"? What information are you trying to get out of the data?
Interesting, thanks for the heads up. I had seen the term 'online' bandied about before, but hadn't realised the technical significance (I thought it literally meant 'can haz internetz').
+1 Welcome to our site and thank you for a clear and informative explanation!
@user1140126 It looks like you have ~ 16 instances of the positive class.  At the points where these instances fall on your predicted probabilities, you will get a vertical jump in the ROC curve.  If you have several instances with similar probabilities, you will get a bunch of vertical jumps, which may cross the line.
I am not really sure. I think that you restrict the permutations to a limited number of records you don't consider all possible permutations and thus you don't consider the full distribution of the feature importance. In case, how would you detect the trend changing the permutation ratios?
Could you instead change the original formulation to: $Y_i=(ax_i+b)(e^{\\epsilon_i})$, where $\\epsilon=N(0,\\sigma^2)$? Or $Y_i=(ax_i+b)(1+\\epsilon_i)$?
In both cases I'd suggest taking a peek at a plot of $y^*=y/(1-x)$ vs $x^*=1-x$ and seeing how that looks.
Just edited the text to make my meaning clearer.
If you have a square symmetric matrix of squared euclidean distances and you perform "double centering" operation on it then you get the matrix of the scalar products which would be observed when you put the origin od the euclidean space in the centre of your configuration of objects. These scalar products _are_ angle-type similarities. They are much like _covariances_. They are not bound within range 0-1, they can be negative, positive, and diagonal elements are not necessarily 1. Still, they are similarities.
I'm not sure we have $Z > 0$. There's always some probability that the $\\epsilon_i$ are both extremely negative, no?
(ctd) ... or see some of the details [here](https://www.amstat.org/sections/srms/proceedings/y2008/Files/300992.pdf). Finally, you might try simulation or bootstrapping.
Been there hacked that. Unfortunately, a simple moving average seems to nail a useable signal better than the Kalman filter in this, and many other examples -- Kalman: [link](http://www.houstonwelsh.com/tmp/kalman20120204.PNG) , SMA: [link](http://www.houstonwelsh.com/tmp/sma20120204.PNG) The data is stationary, is so far as a Dickey Fuller p < 0.01. Maybe I'm doing it wrong. A script to run a particle filter on this data, and other pairs trading candidates would be awesome (I think).
Take a closer look at the output of your favorite regression command: it should report an $F$ statistic and a p-value for it.  That is also the p-value for the $R^2$, because $F$ and $R^2$ are directly and monotonically related. For ordinary regression with $n$ data, $F = (n-2)R^2/(1-R^2)$. Its p-value will be the p-value for the slope. Therefore if you have ever used a p-value for $b$ in ordinary regression, you have used a p-value for $R^2$.
You could shorten the names considerably if you use (1) variable labels and/or (2) a meta-data document.
Try to imagine the data-generating process. When do people usually listen to the radio? The shortest answer is it depends - different groups have different patterns, different waking and sleeping hours; thus you will most likely have a mixture of distributions. I don't quite get your constraint on the maximum number of users.
In your case you essentially have 1 value of AUC for both models. Not much can be done with that except compare them on face value. You are better off I think using a bootstrap or k-fold or other type of resampling and then use a paired t-test or  Wilcoxon signed-rank test
Also: $11300 \\cdot 21500 \\cdot 8 = 1\\,943\\,600\\,000$, so even the most compact of storage is going to take up 2GB of memory just to store the matrix. And, anyway, there is no way that $X^T X$ is invertible in the first place if you have 11K observations and 21K features.
(2)If The first and second model were based on sample,say,$\\hat Y_t=\\hat\\beta_0+\\hat\\beta_1X_{1t}+\\hat\\beta_2X_{2t}\\ldots+\\hat\\beta_pX_{pt}+e_t \\ldots(1)$, then estimated beta parameters depends on $X$'s. So if the case is of `sample` i can't take those estimated beta parameters  as common factors.Isn't it?
Ok; for the future, you can ask mod to do this by flagging question.
Sounds like you want a 'hash function': http://en.wikipedia.org/wiki/Hash_function 
This is in the realms of computer science rather than statistics though, so I'm not sure it belongs on CrossValidated.
Alecos, I think a little more caution about completely solving the question is in order for what appears to be a [self-study](http://stats.stackexchange.com/tags/self-study/info) question (even if the OP hasn't yet tagged it that way). That is, it looks like you're 'doing the OP's homework', in detail.
Continuous random factors don't make sense to me. See Douglas Bates, [_lme4: Mixed-effects modeling with R_](http://lme4.r-forge.r-project.org/lMMwR/lrgprt.pdf), pages 1-2: random (or fixed) effects parameters are associated with the levels of _categorical_ covariates.
Although this is a nice idea and well-explained, it would help to see some justification for why comparing *medians* is appropriate when a comparison of *means* is desired.
Hi, I used the following command: zelig(choice ~ Deviance + Votes, data = data, model = "mlogit")
Because these issues have been discussed extensively on this site, I will be brief: (1) the p-value is a probability *conditional* on assuming the null hypothesis (it's not some unconditional "chance" *and it's likely contrafactual*) and (2) it is not a chance of "the result you're seeing"--which in this case is practically zero--but rather it's the chance--under the null hypothesis--that your result would fall within a "critical region" for the hypothesis test. Although this may seem like nit-picking, much confusion can arise from misinterpreting loose language.
Having taken a look at your data, I don't think you can see any sort of pattern or dependence with any of these methods. You'd have to collect another two-three-better five hundred points!
Honestly, when I come across references that use miscegenated phrases like "reverse-engineering a formula" and have godawful typsetting for symbols, the first thing I do is look for a different reference, because you can be pretty sure the writer either doesn't really know what they're talking about or they have little idea of how to communicate it well.
@Shanker, if you want to fix the coefficient at $1$, then you don't want it in the model - you just want to add the corresponding variable to the right hand side of the equation, which is what offset does.
As I mentioned in main question, if i want bigger companies based on `log(total assets)` besides higher probability of bankruptcy, what should i do?
I wonder if applying this method using a large number of bootstrap iterations would provide a reasonably valid outcome?
If you know the answer, please post it here; you can help other people that may stumble upon a similar problem. And earn some reputation.
I know of several definitions of "Gaussian processes," so it's not apparent what your question is really asking about. But as you consider how to clarify it, ask yourself this: exactly how would you parametrize the Gaussian process you have in mind? If you cannot do it in a natural way with a finite number of real parameters, then it should be considered nonparametric.
Right, you are the enthusiast of the minimal sample size requirements... Firth's correction works with i.i.d. data only. You can believe in anything, but you'd be better off running some simulations to see if any given belief is justified in a given data situation. With a perfectly balanced data set and continuous reponse, it MAY work OK. With a heavily imbalanced data set, in terms of response, you see only a far left tail of the normal distribution of the random effects, and you are willing to bet on this tail being well approximated by one-point Laplace in `*lmer`??? :-\\
+1 for the tip about `cex`, but I think the OP wants all stuff on the same plotting region, not on separate ones.
For anyone interested, I found this based on Markov Chains... it uses the weather as an example. http://www.jstor.org/pss/2533067
120 variables is not a high number for a million observations. Multicollinearity will be a problem only when the standard errors of the estimates become too large to be useful--and the large number of observations will be a great help in reducing all the SEs. One might start, then, very simply by estimating the full model and inspecting the SEs: maybe they will show there isn't any problem at all to deal with.
On the contrary, the log likelihood function is perfectly well defined for all finite values of the parameters, even with perfect separation. The difficulty here may be mainly a matter of clarifying one's meaning: I think you may be trying to refer to the value of the log likelihood at the optimal parameter values.  Since there are no optimal parameter values in a case of perfect separation--as @Juho points out--a numerical search algorithm can run into problems.
Many thanks for responses - I will look into suggestions made
(+1) Homoskedasticity is an assumption in the sense that $X_1,\\ldots,X_n$ are assumed to be i.i.d. (and thus have the same variance). It's possible that this is what the OP meant.
+1 I could not think of the name of this before; I was going to come back and google for it just now - I am glad you came in with that answer.
possible duplicate of [Repeated measures ANOVAs instead of repeated measures MANOVA w/ small N?](http://stats.stackexchange.com/questions/34911/repeated-measures-anovas-instead-of-repeated-measures-manova-w-small-n)
That's a wonderful answer ! Thanks a lot for your thoughts !!. I like the way you answered my concerns about applying Gini co-efficient to my problem. I will explore the possibilities of applying multinomial distribution here.
What is the objective of your modeling?
Thank you for sharing the papers.
I think your answer resemble the k-fold idea, ;-)
Maybe you shouldn't take the definition of "information gain" quite so literally.
Could you also explain why a Frequentist multi-level model is not as straightforward conceptually?
http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2013.html
see the graphical models lecture in the above link for a (short) technical introduction to bayesian networks. it explains most things. 

if you are using a model that iterates, i.e. the parameters change, then your prior will not be too important as long as it is not degenerate in some sense (i.e. a dirac spike, say). a prior that's analytical (say the standard Normal distribution) makes sense in a lot of cases. 

you should see which conditional relationships hold from the data. then you can model that into the bayesian net
An alternative way to look at the more general problem:
On round 1 put every ball in a random bucket.
On round 2, keep the 1st ball that went in each bucket in the same one. Move the second one across to the next bucket. Move the third one across by two buckets, etc.
On round 3, keep the 1st ball in each bucket in the same one. The second ball (from the 1st day) now must be moved across by two places, the third ball has to be moved across 4 places, etc.

A general formula for the places moved ith ball on the rth round would be:
(r-1)(i-1)
`s3d <- scatterplot3d(x.mds$points[,1:3], color=as.numeric(gl(2,50)))` would highlight the first 50 points in black, the remaining ones in red; you can pass any logical vector or factor to suit your data. HTH
At the very least, as indicated in the equation above, this would not be a stationary effect.  I would think that a first step would be to bin the readings according to time interval and then compare them.  I don't know how many readings you have but this comparison could be run through something like [Pearson's Distribution](http://stats.stackexchange.com/a/72434/31323) as a starting point - to try to classify the nature of the distribution.
"If I believe solely in F as my prior distribution, then no data should convince me otherwise." This is the antithesis of Bayesian inference, which is more along the lines of *take what you believe in one hand and the world in the other hand, and mush them together and see what come out.* Wash, rinse, repeat.
Yes, I am sorry. I read that as implying something it didn't; my apologies. I was primed by another answer which did say it (at least in a comment underneath, and I took you to be agreeing with the sentiment expressed there about the denominator when you pointed at the answer), so I got the impression you were trying to say much the same thing here.  I will delete the previous comment as irrelevant.
If X's (independent variables) are measured variables, then it is a "random variable" instead of a "fixed value". According to Wikipedia (http://en.wikipedia.org/wiki/Random_variable):
"In probability and statistics, a random variable or stochastic variable is, roughly speaking, a variable whose value results from a measurement on some type of random process." Am I correct?
Why don't you write out the formula explicitly for the cases $n=2,3,4$ and see if it is correct? Then use induction to prove it in general.