tttjjj/elt_bench huggingface dataset:
One reason I offered a more elaborate answer is that this definition of "parameter" is not good enough. For one counterexample see [my comment to @gung's answer](http://stats.stackexchange.com/questions/63386/is-any-quantitative-property-of-the-population-a-parameter#comment122123_63387). Intuitively, a set of *parameterized* distributions forms a finite-dimensional topological manifold-with-boundary; a parameter has to be a *continuous* function defined on the manifold. This is more than just a technical requirement, because it relates to the sampling distributions of estimates.
Sure, you could call it that.  The function you asked about is itself a normalization, but the normalization fails if any one of the exp(x) is inf!
Just one more time, anyone?
Yes, I'm sorry for not asking the right answer. Are there any strategies to ensure the stability? Seems giving the iteration a large value is the only way I can think of.
Stata does have a [soundex](http://en.wikipedia.org/wiki/Soundex) function for help with stuff like this. In my experience, though, one's best friend is a string processing oriented tool like PERL or AWK, rather than the limited data cleaning capabilities built into statistical systems. (John Chambers, one of the originators of the `S` system, advocates PERL for such uses.)
Welcome to this site! A working example or short illustration would be highly appreciated.
Welcome to Cross Validated! Could you please explain a little more what you're trying to do? AIC is calculated from the likelihood of fitted models & the number of parameters, & the difference in AIC between models is used to compare their likely predictive ability; I'm not sure what these 'functions' are, or what the difference between 'algebraic' & 'absolute' values is.
Thanks for the additional points and for sharing your perspective.
By "no information" I'm referring to the "relevance" problem pointed out by Charles Y. Zheng below.  Asking questions like "What is the third letter of your last name?"  "What is the 4th digit of your phone number?" etc.  Likewise, if the questionnaire is relevant and optimal for a given population, there should be no most-likely answer to any question with or without knowledge of the irrelevant questionnaire.
This is an *enormous* subject. I can see that narrowing your question might be difficult, but any adequate answer would--and has--taken several large books.  You might begin your research by linking through the [tag:model-selection] tag I added; it will show you some threads in which these topics have been discussed.
Also thanks @whuber for the info.
If you have missing cells you cannot use ANOVA. Unbalanced just means that the groups have a different size. Are there really cells that are empty?
Do you have time series / panel data?
@Яaffael I am talking about non-parametric bootstrapping.  You take your sample (say of size 100), re-sample from it with replacement (100 times yielding a bootstrap sample of size 100), and then re-calculate your estimator of interest. You are treating the sample as a toy population, simulating drawing a sample from it, calculating an estimator.

If you sampled from the toy population without replacement, you would exactly copy the toy population in the sample, getting the original estimate as the new estimate (ie variance=0).  To avoid this, so you sample with replacement.
What definitions and relevant theorems do you know about sufficient statistics?  How have you tried to use them to tackle these problems?
I would suggest performing the regression without worrying about whether `x` is approximately Normal. Forcing it into that straightjacket would be an error in almost any application.  Besides, due to the many ties, your `x` cannot be transformed to look even remotely normal.
A related approach can be found in Cook et al. (2006; http://www.stat.columbia.edu/~gelman/research/published/Cook_Software_Validation.pdf). I have used Cook et al.'s approach on two occasions, and I have been impressed with the results. I have not used Geweke's approach, but according to Cook et al., "Geweke’s approach has the advantage that only one replication needs to be performed...A disadvantage is that it requires altering the software to be tested." They also say Geweke's approach requires priors with finite variance, whereas their's does not.
Do you average out the Res from each cross-validation run? Do you actually calculate Res from the residuals of the hold-out samples?
Ok so you have a set of functions that define the [relative] density for any given point in the space. For rejection sampling to work you will need the densities to be normalized (for the purposes of this explanation let's say we determine the maximum density observed across the space). So my suggestion then is to generate coordinates in the space randomly (and uniformly distributed), for each generated point generate another uniform random value V in the range [0, maxDensity]. If V is <= the density described by your function then accept the point as a valid sample, otherwise reject it.
Larger sample sizes will decrease the probability of similar sample means if the population means aren't the same.
OMG!this is so helpful! Thanks Rioraider. I cant believe that I could not this figure out because I have been working on it for almost a year and didn't realize this. Thanks again for answering.
+1: Not only is this suggestion a good approach to the problem in question, it shows that throwing away variables is not always the right (or even a good) approach to solving collinearity problems.
what are X1 and X2?...can you make this self contained?
@gmacfarlane: I've tried to simulate data where MNL would be better than series of binary logistic regressions, but every time on average the quality was the same. I was comparing lift charts and after averaging results from few simulations they looke almost the same. Maybe You have an idea how to generate data so MNL beats binary logistic regressions? Although MNL had a great advantage, its scores could be interpreted as probability.
@Sam The chi-square statistic is a measure of association of the two variables ("attribute" and renewing).  When there is no association, the chi-square statistic should act like the 500 renewers occurred at random in the population of buyers, regardless of the values of "attribute."  In this model, the number 3E-111 estimates the chance that chi-square could be as large as it turned out to be.  You can do all the calculations in Excel, although stats software like R (free), Stata, SAS, JMP, Mystat (free), etc., is easier and more reliable.
Could you further elaborate?
I was referring to the coefficients. From an economical point of view the coefficients should take on certain levels that "make sense". I fear that my model somehow overestimates the effect of xi,t−xi,t−1 (bookvalues) these values and thus the coefficients are way off values you would expect.
Also as mentioned in my post, I am not sure if multivariate t distribution satisfies it or not.
Great answer -- exactly what I was looking for.
(+1) The first paragraph quickly gets to the heart of the matter.
I suggest you put this comment into the main text of the question.
yes, you could have a Gaussian mixture model having two modes whereby you "spread it out" by moving the modes away from each other while reducing the variance of each mode.  The overall variance goes up, but the entropy goes down.
I don't think this is quite the same thing as ecological regression - that's about trying to infer something about the within-group slopes when you only have the group means. Here you have the within-group slopes.
How about a metric like Mean Squared Error between actual vs predicted values?
"What model should I use? What are the steps needed? & How should I check the quality of the model?" is too broad to be answerable. Way too broad to be answerable. Such questions cannot be answered reasonably in a format like this. You will need to take several statistics classes to develop an adequate understanding of these issues. In the interim, we would be happy to help you with suitably focused questions.
Another way to motivate this is to re-write the loss function as a (weighted) sum of the width of the interval plus the distance, if any, by which the interval fails to cover the true $\\theta$.
Define 'strong sources', please. We could certainly measure the observation (though it could also be described other ways), but we estimate the parameter.
$c$ should not affect the result as $\\frac{\\sigma_x}{\\bar{x}}$ does not change if all the values are multiplied by any positive constant $k$.
Yes, my data is scaled and centered and orthogonalised using pca. My data has 60 columns and 100k observations with binary output. I have observed very slow improvements in out of sample prediction accuracy (A logistic model trained using glmnet is outperforming the perceptron). I am using a single layer with 300 fully connected hidden units and sigmoid(tanh) activation.
(1) Where does the one degree of freedom in (4) come from?  (2) The complementary CDF for $\\chi^2(2)$ is simply $x\\to \\exp(-x/2)$.
As written, some of these issues might be a better fit to a different forum, perhaps one related to investment (such as questions about what's 'standard'). Additionally, you shouldn't expect there to be a single best measure unless you specify some criterion by which to judge 'best'.
Mainly The Moment Generating Function Uniquely Determines the Distribution.
See my comments to the other answer, which may (or may not) help explain some of the discrepancies you note.
What Nathan is describing is also the basis for Bayesian methods of significance testing.  I have used (and currently use) the [Bayesian Estimation Supersedes the T-Test](http://www.indiana.edu/~kruschke/BEST/BEST.pdf) (BEST) approach.  You should look at that framework if you intend on implementing a pooling approach.
This reads like a routine question from a textbook. Is this for some class?
Did you leave off something on the end of your question?
This question appears to be off-topic because it is about how to make a plot in R.
I'm a big fan of the Gelman/Hill book.
The tough thing about conditional probability is that many times these quotients do not exist, because $P[B]=0$, but nevertheless the conditional probability $P[A|B]$ makes good sense. Many rigorous treatments lead back to a fundamental idea, the [Radon-Nikodym theorem](http://en.wikipedia.org/wiki/Radon%E2%80%93Nikodym_theorem).
Please, keep us informed on your progress with your related question on SO, http://stackoverflow.com/q/8972244/420055.
Here are two of them that I thought of:
1) number of attempts in the 10th trial
2) The difference between number of attempts in the 10th and the 1st trials
Applying an arma structure to the residuals would I think be a little different from the result the gls gives regarding the regression parameters.
For ordered categories, which I expect "opinion" will be, you might use something like [this](http://i.stack.imgur.com/yJhWA.png) or [this](http://i.stack.imgur.com/U7a4i.png) ... or a pair of barcharts, perhaps.
@MYaseen208: Are you looking for the convex hull of your data?
Is there any method to rank the outlyingness from result of hclust? Since the dataset is large, it is hard to see outliers from just looking at the Dendrogram.
Beeing more precise
Basically, I have a dataset of 10 years. So, over these 10 years, the occurence of an edge between the two nodes will be given by 1( if an edge is there in that year ) and 0 ( if an edge is not there in that particular year ). So, if I find a pattern in that sequence, I come up with this output file.
I took my best guesses as to what you mean and provided some information and links related to the slope, p-value and $R^2$.
@Adidul : You could upvote or even accept Aniko's answer if you are happy with it! :D
Note that the numbers which come out of Probit and Logit models look as if they measure roughly the same thing, but are often different numerically.  When you translate them back into real life, the difference between the two usually becomes much smaller.
I ordered Statistics in Plain English... I'll accept your answer when I get it on Thursday if I dig it as much as I think I will.
Comparing the R packages in the [time series](http://cran.r-project.org/web/views/TimeSeries.html) & [econometrics](http://cran.r-project.org/web/views/Econometrics.html) task views to the Eviews [feature list](http://www.eviews.com/EViews8/ev8features.html) should help you. Note that R describes itself as a language & environment for statistical computing & graphics - if you know an algorithm you want to implement, & someone hasn't already done so; you can, building on functions & data types already available.
Do you mean geometric interpretations like those offered in http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues?
I'd fell better if $p=2\\text{Pr}(\\chi_1^2>1.75)=0.1859$ ;-)
"just focus on the question at hand instead of inquiring further and further into the motivations behind why". I'm sorry; we're not just being curious or trying to be awkward, honest. Statisticians learn, often the hard way, that it's usually a very bad idea to tell clients how to do something without first enquiring why they want to do it.
There is a legitimate question of whether 'variance explained' is really a meaningful quantity to wonder about for GLiMs, you may want to read through this thread: [Which pseudo-$R^2$ to report for logistic regression](http://stats.stackexchange.com/q/3559/7290).
Welcome to the site, @TobiasUrff. This isn't really an answer to the OP's question. It seems to be a set of questions for clarification. Please only use the "Your Answer" field to provide answers. I recognize it's frustrating, but you will be able to comment anywhere when your reputation >50. Since you are new here, you may want to read our [tour page](http://stats.stackexchange.com/about), which contains information for new users.
Note that you should only expect the $z$'s to be normally distributed and the p-values to be uniform if you know that your null hypothesis is true. In the data you have, the $z$'s can have any distribution. To check your procedure, you should really simulate data where you know the null hypothesis holds, apply your test, and check that resulting distribution of p-values is uniform.
Sounds rather  like homework; if so give it the homework tag.
I agree that it is a good idea to look at the distribution of the waiting times. A Lilliefors test against an exponential could help to confirm your process is Poisson. But if you do have data with censored times at the end, I still think the approach above is the best way to estimate the $\\lambda$'s.
I disagree, Jaynes' book is a terrible recommendation in this circumstance: 1) the notation is sloppy and non-standard, which makes it difficult to cross reference with other sources, 2) he's long winded and gets bogged down in silly and irrelevant arguments (the OP asked for the "shortest route") 3) there's also the errors (such as the marginalization paradox)
@whuber: Indeed.  Offtopic, but for anyone interested I have an interactive selection of world map projections [here](http://www.btinternet.com/~se16/js/mapproj.htm), some of which are equal area
Another note is that I am not getting these problems when just using `logit` (no panel data), and random effects.
Signature removed. If you want your posts signed "Johnathan", you change your username from `user43502` to `Johnathan` and it will be appended to the bottom of every post you make.
Yes I realize this but I was hoping to do it in R because of the flexibility.  You could alter all sorts of parameters to better represent the data that you can't with Many Eyes.
If A should be the same: why not run a single regression on pooled data?
The correlation between $X_i$ and $X_j$ is given
$$
Cor(X_i,X_j) = \\frac{Cov(X_i, X_j)}{sd(X_i)sd(X_j)}.
$$ If your correlation matrix is V this is
$$
Cor(X_i,X_j) = \\frac{V_{ij}}{\\sqrt{V_{ii}}\\sqrt{V_{jj}}}.
$$ Maybe this can help you set up your covariance matrix, especially if you are able to simplify your problem by standardizing each variable.
If the gradient in log-hyper-parameter space is negative, that is no problem as it doesn't matter if the log-hyper-parameters are negative, once the exponential is taken the hyper-paremeters themselves are guaranteed to be positive as exp(x) > 0 for all x.
Could you clarify what is meant by "the integral .... exists"? There seem to be two different issues here, viz. (i) does the integral $$\\int \\frac{f_{Y\\mid X}(y\\mid x)}{f_{X\\mid Y}(x\\mid y)}dy$$ exist or not? and (ii) if the integral exists, is its value $\\frac{1}{f_X(x)}$? Or are you saying that whenever $X$ and $Y$ have conditional densities such that $$\\int \\frac{f_{Y\\mid X}(y\\mid x)}{f_{X\\mid Y}(x\\mid y)}dy$$ exists, then it must be that the value of the integral is $\\frac{1}{f_X(x)}$?
How about PCA/FA? You would shrink correlated variables into factors and work from there...
The only way out seems to be an application of the Kalman filter as e.g. in http://home.ubalt.edu/ntsbarsh/stat-data/cardamone.pdf
I modified my answer.
I am very impressed with the quality of Statistical Learning with Applications in R. I think I will give it a try and possibly buy it. Thank you.
Thank you @whuber. As always, it's a real pleasure to read and learn from your excellent responses. Many thanks!
Why dont you just find the square instead of using SVM?
One way to think about it: In a standard logit model, you don't have two equations, one for the "0" group and one for the "1" group; you just model the "1" group relative to the "0" group.
Actually as I re-read your question, this probably wouldn't support the action you need on row x+2. "add column title and data type".
Have you explored *directly* the relationships between $S,B,N,M$ and the dependent variable?
Note that you don't need to evaluate the probabilities to answer the question, you only need that –3/4 < –2/3.
I'd add that lack of experience is also a reason __to__ write an R package. Writing a package for the first time is not only fun and a challenge, but it actually helps one formulate ideas about how to design a 'proper' package that will be useful to oneself and the community. In other words, even if one lacks experience, it's still a good idea to write a package in order to get experience in doing it.
How exactly to get percentiles depends on what software you have, but normality is irrelevant and you would not use a normal distribution table in any case.  Rank your data. Then the highest score is 99% tile.  If you had exactly 100 cases, all with different scores, then each person would be one percentile lower. Since you probably have some odd number of cases, you have to do the calculation or get a computer to do it.
@caracal, `P <- X %*% solve(t(X) %*% X) %*% t(X)` doesn't produce r=0.6, so that's not the work-around. I'm still confused. (I'd be happy to mimic your expression `Q <- qr.Q(qr(Xctr[ , 1, drop=FALSE]))` in SPSS but don't know how.)
By the way, I'm also struggling to find a formalization for the approach I described. If I knew the formal term, it would also help me with my research. Maybe there is a variation to this algorithm that addresses at least the independent/irrelevant dimension issue.
Hovius sorry I was trying to find the links while typing.
Finite mixture yes, polluted I just meant the distributions are based on a combination of multiple variables, each distribution has its own mix of variables. I wanted to normalize them all to each other, I found that ecdf did just that (except for skewed distributions...)
I am trying to determine the maximum possible systematic disagreement two raters/annotators could possible have achieved. In a contingency matrix, disagreement is represented by the off-diagonal cells. With 'repopulate' I mean the permutation of the existing cell counts such that those three conditions are met. I can then relate the Chi square of the contingency table with observed counts to the maximal Chi square, and get an idea of how much of the disagreement isn't random disagreement.
And of course, 17 is the least random number.  http://www.catb.org/~esr/jargon/html/R/random-numbers.html  but my favorite random number is 37: http://jtauber.com/blog/2004/07/09/37_is_a_psychologically_random_number/  (though, also see http://scienceblogs.com/cognitivedaily/2007/02/is_17_the_most_random_number.php )
The comparison to zero might require some testing. An alternative would be something like `abs(sigma) >= .Machine$double.eps`. The issue with `zapsmall()` is that it only sets very small numbers to 0 when there also is a big one. A hypothetical example: `a <- c(.Machine$double.eps / 10, .Machine$double.eps / 10); b <- zapsmall(a); b[b != 0]` gives both numbers that are numerically indistuingishable from zero.
Have you ever heard of "mediation analysis"?
@tomka: yes everything is. I have provided the R code used for the computation so that it is clearer.
What did you end up with -- could you put up a picture or two ?  Also, "setting all values to zero except for one predictor" -- don't you want the gradient around the current best values, not around all 0 ?
@Scortchi Normalizing the predictors is not necessary, but can make interpreting the coefficients from the regression easier: After normalization, big coefficients correspond to important predictors. Also, without normalization, the coefficients of interaction terms can be seriously misleading. That being said, normalization won't affect the predictions you get from your model, so normalization only matters if you intend to interpret the coefficients in the regression.
As an additional contribution, specifically regarding the $L_0$ norm, I don't know that I would say it is because it isn't "mathematically flexible"; I think it is primarily because the optimization is prohibitively expensive (there are ways of trying to do it, but I don't think anything works in complete generality). I know of one "big-cheese" figure who works in variable selection who said he would love to use an $L_0$ penalty and that computation is the only reason he doesn't.
@Zen: P.S. #2 By "$\\Pi\\left(D\\cap E\\right)$" i meant "$\\Pi\\left(D\\cap\\left(A\\times E\\right)\\right)$".
@Tomas, the N[i] you generated are just initial values, and the crash occurs when a posterior simulation of N[i] is zero
After eight questions, don't you think it's time to (a) register yourself and (b) mark some of the previous answers as accepted?  Don't wear out your welcome.
@whuber: thanks - I clarified the answer to point this out more strongly and also that this is one extreme case.
@Omar: different things being different input variables, i.e. you want to do a variable selection? Can you explain your application a bit more in the question?
@gung I think that the samples may well have different means, it does not bother me per say unless it means that I can't calculate variance for a sample across conditions.
(+1) And, here, I was going to say the opposite. When I first saw the title of this question I thought: *Hype optimization? No, I think they're already doing that just fine!*
Thanks, I will try this one. Found an implementation together with the Levenberg-Marquardt algorithm.
A good guide to questions of this nature is to ask, do you want to confine your conclusions to the sample (those five seminars) or do you want to extend your inference to the population?
Thanks for that. Very useful.
I think that what @cardinal is getting at is that there are some games with positive expected winnings in which the player nonetheless has a non-trivial chance of losing a lot of money.  One needs to consider the entire distribution of possible outcomes, not just the mean winnings, to make rational decisions.  Especially since most people find losses hurt more than equal size gains feel good.
I meant L-kurtosis, but somehow wrote L-skewness...