tttjjj/elt_bench huggingface dataset:
@Statsnewb Update: According to Spanos' "Statistical Foundations of Econometric Modelling" (1986), Slutsky's 1927 paper "The Summation of Random Causes as a Source of Cyclical Processes" gave rise to the moving average (MA) model. That said, I doesn't seem to be the case that this is the source of the term "moving average" since Slutsky uses the term "moving summation". One step closer to finding this one out! :)
@Zloy Sorry, I'm not familiar with what kind of neural net packages are available on matlab.  Here is a tutorial from Andrew Ng's class on sparse autoencoders.  http://www.stanford.edu/class/cs294a/sparseAutoencoder.pdf
This question appears to be an update (i.e., duplicate) of your previous question: http://stats.stackexchange.com/questions/23053/  In general, it is best to edit your existing question to update with new information rather than asking a new question.  Editing will bump your question back up to the top of the queue & let people know that the answers you've gotten so far aren't what you needed, so it accomplishes your goal.
If I'm understanding you correctly, this is analogous to having several choices to sample from. At each round you select a choice to sample from. The choice you selected will yield a reward. You want to maximize this reward. Each choice has an unknown distribution and parameters. This is the multi-armed bandit problem, which is essentially solved. If you have side information about each choice, this is the contextual bandit problem.
Griffis & Stedinger (2007) use the term pseudo, because the way how calculate the sum of squares is not the traditional one as that used in OLS.
@Marcus, I find your usage of term distribution a bit confusing. P(X has this much wealth)  is a number, $P(X<x)$ for given $x$ is  a distribution of $X$. Do you per chance need something like [Lorenz curve](http://en.wikipedia.org/wiki/Lorenz_curve)?
I can't follow the logic here at all.
@Jacki, Your experimental design looks like a special case of Split plot design, but to decide the special case, I need extra details. Can you please represent your design inform of a picture (see my update) ? Currently, I don't see how you use the varieties? Is the difference between varieties of interest or not?
The only awkward part seems to that you think product choices depend on products, their categories *and the choice in the other category*.  Perhaps you'd like to elaborate a bit on this last dependency?  Is it an ordered choice / framing effect?  Or a cross-covariate constraint (e.g. get 1500 calories for as close to $10 as possible, or get one item from each major food group),   or some other thing you have in mind.
@Zen Indeed (in fact it _increases_ with sample size, although boundedly), that's why Cardinal's bound is more useful for most sample sizes.
I have edited the question to give you the exact context. Does that help? Thanks.
@ Jason thanks :-) i will do that
I mean take the differences $\\hat\\theta_1-\\hat\\theta_2$. Within a bootstrap iterate, the computation is actually like an unpaired test, not a paired test ... indeed it's exactly analogous to the numerator of a two-sample t-statistic (where in that case, $\\hat \\theta_i$ is a sample mean). ... though if you mean 'paired' in the sense that we're pairing by bootstrap iterate, then yes, that's exactly what I mean, but it's not really pairing in the usual sense, since the two are quite independent. It's not necessary to do it this way, but it's certainly convenient.
@JohnMoeller: sorry, I completely misunderstood that (coming from a field where "a sample" is a physical specimen of some sort).
You and the OP are using the term *sparse* in two very different ways. The paper you reference is not at all geared towards handling *sparse matrices*. It is concerned with taking a *dense* $n \\times p$ matrix and finding a smaller $q < p$ set of features (columns) to cluster the $n$ data points. The penalization approach uses $\\ell_1$ and $\\ell_2$ ball constraints as well as nonnegativity constraints and, as far as I know, appears to have only been tried on problems up to a couple orders of magnitude smaller than the the OP's.
A good list of very closely related threads can be found with [a suitable search of this site](http://stats.stackexchange.com/search?q=dummy+categorical+intercept+reference).
I'm sorry, how can you have "paired data" of different sizes??
@ttnphns: I don't have a better word for structural variables (time and space variables). The differences between them and the other independent variables (the input variables) is that the model code doesn't care about where in time and space it is running the calculations - it just cares about the input variables.
Thank you :) I did plot my rf1 but apart from QQ none of these plots seems familiar to me. As you say, the 1/0 class is imbalanced (like 15/1). In the next step I'll try to apply the connection for the intercept as I assume the prevalance in entire population would be similiar to the one of the sample. I'd also try weighing my observations. I'll let you how it affects residuals' distribution.
@Nick Cox Time is the independent variable. The model should read $y=at^b + c$
I have a SPSS macro. May send it to you if you leave your email.
Are you doing logistic regression, or linear regression, or something else?
The OP says the figure is attached but there is no figure here. It seems to be lost during the editing.
Hmm, two down votes.  If the answer is so bad it would be nice have some commentary.
How do you plan on taking logarithms of any counts that are zero?
Is d a parameter of the logistic regression or a predictor used in logistic regression? It sounds like a predictor but you call it a parameter. If it is a predictor, then its domain is irrelevant.
thank you very much for the help. Actually I'm not against Grid search, I wast just wondering if there is other better solution. But your reply gives me a clear idea.
Your question is vague and would benefit from clarification. In what context are you using a ROC curve? What do you mean by goodness of fit in this situation? Edit your question and include a concrete example of your problem. Try reading the three papers I suggested below to better organize your thoughts.
I am not aware of any claim that RBF SVM is similar to a K-NN classifier (it certainly isn't going to be equivalent).  Can you give a reference to a place where such a claim has been made?
Thank you very much peter,
One thing I noticed that the variances in your original example are very small 6.5e-6 = 0.00000065. It depends a bit on how you scaled your variables, but to me this suggest that either you need to rethink the scale of your variables or your variance is _de facto_ zero
I wanted to ask a similar question. Did you try to contact the authors? It often works.
@DilipSarwate As an extreme estimator, the probability limit of MLE $\\hat{\\theta}$ is the solution of the minimization problem $\\min_{\\theta} E_0(\\log f(x;\\theta)-\\log f(x;\\theta_0))$.
tall=many pts? fat=many features? is this std terminology anywhere, used in refs somewhere?
Hi Whuber, very interesting connections!  once I'll delve into this more, this gives me a lot to look into - Thanks!
@Vass an example of how you would do that, would be appreciated ^^ if you can.
+10000 if I could! Thanks very very much
$m$ accounts for $c$, but it is not that important.
@IrishStat Basically, DBSCAN clusters points in n-dimensional space. So its suitability for time series analysis is very limited. It works for my special case since I expect only a single cluster in my (1-dimensional) histogram. To answer your questions: When using appropriate parameters, the value 5 of your example would be detected as outlier. I suppose it can deal with trends, but in the time series '1, 2, 3, 4, 5, 6, 7, 2', the value 2 would not be regarded as outlier! I hope, that gives a rough idea about the capabilities and limitations of DBSCAN.
Related questions: http://stats.stackexchange.com/questions/7959/algorithm-to-dynamically-monitor-quantiles, http://stats.stackexchange.com/questions/7251/how-to-make-a-combination-aggregation-of-quantile-forecast, http://stats.stackexchange.com/questions/14006/distributed-quantile-algorithm-with-duplicates.  They all contain potentially useful information (even though not all are answered).
Yes, doesn't make a difference in this case. The difference in intercept and all the problems are the same for two or more groups. The interpretation of the 0 is the issue. And independent of the number of groups, with treatment coding it is the first group whereas for effect coding it is the unweighted grand mean. This affects all higher order effects. And now give me my upvote...
Yes, thank you, I'll try that.
(+1) http://en.wikipedia.org/wiki/Distance_correlation might offer some useful ideas.  Also check out the material on our site about [copulas](http://stats.stackexchange.com/search?q=copula).
Note that $W=F_X(X)\\in [0,1]$, and you proved that $P(W\\leq t)=F_W(t)=t$. Hence, $W\\sim \\mathrm{U}[0,1]$.
Thanks. Is it possible to use the fact that the integral of a pdf has probability 1?
But you understand the general idea, right? The desired conditional density is zero outsided the conditioning set A, and proportional to the original density inside $A$, but we renormalize by $P(A)$ (otherwise, $P(X\\in \\mathbb{R}^n\\mid A)$ would not be equal to one).
Just as a general practical principle, when you are performing a bootstrap by means of simulation--or more generally when you are approximating any p-value--you should favor a *conservative* decision rule. This follows from the asymmetry in possible conclusions: "accepting" the null is more tentative than rejecting it. Rejection is an assertion that chance would be an inadequate explanation of the observed differences, whereas quite a bit of the distinction between $p\\lt\\alpha$ and $p\\le\\alpha$ is due to the operation of chance in your procedure.
Also, your tests cannot probe the hidden variables that are a consequence of ABC.
@Keith Which topic in particular?  Regression?  Time series?  Adjusting data? Model selection?  Political statistics?  It would be good to have *some* place to start in following your suggestion!
Your tables are very hard to read.
OLS does not needs the outcome to be normal, it does, however, assume the *residual* to be normal, or the standard error can be biased, leading to a wrong p-value. Some OLS models using count outcome tend to have non-normal residual (esp. when counts are low or zeroes are abundant,) and hence using Poisson family to model the error is a better choice.
Did I get the right link for `Orstat`?
Was it explained why skewness should be eliminated?
I just recalled an excellent blog post on a related issue, maybe this helps: http://darrenjw.wordpress.com/2012/06/04/metropolis-hastings-mcmc-when-the-proposal-and-target-have-differing-support/
Bayesian or non-bayesian is not an issue here. If you fit your regression using maximum likelihood or Bayesian estimation wouldn't affect how you code the dummy variables...
10th decile would be maximum income? Does world bank really give such data, i.e. the maximum income of individual in population of specified country?
most of the articles/discussions that i see about big data have less to do with the analysis, and more to do with the issues of how to properly and safely store petabytes worth of data.  Banko and Brill published an (at the time controversial) article in 2001 that claimed that the actual algorithem used was of little concern, and that they all eventuall got to generally the same error rate.  What mattered was the amount of data used. data.http://acl.ldc.upenn.edu/P/P01/P01-1005.pdf
That's not going to work in one important case: from the manual: 'The default initial estimate is the Minimum Volume Ellipsoid computed with CovMve'. Suppose there are some discrete variables in the design space. MVE's gonna chocke on them. e.g. there are situations where you would really want an convex M-estimator and not a re-descending one.
Great, ill try tomorrow to post a data det so you can see how it works
Any evaluation methodology you use that allows you to get a perfect result must be busted... there is no perfect system, ever. Because there is no perfect data in reality, either!
I consider any result that is close to 1.0 to be fake.
I see nothing in the original question that restricts it to economics and nothing in this answer that addresses the time-related aspect, e.g. why would one log-transform years? (My own perception is that this is unusual; original poster never substantiated the claim any way.)
Maybe the original poster is interested in quantifying the phenomenon.
(+1) Appealing to conditional densities is not strictly necessary here. Note that $\\renewcommand{\\Pr}{\\mathbb{P}}\\Pr(N_t = n) = \\Pr(S_n \\leq t, S_{n+1} > t) = \\Pr(S_n \\leq t, S_n + T_{n+1} > t)$ and we need only integrate the joint density over the region $\\{(s_n, t_{n+1}): 0 \\leq s_n \\leq t, t_{n+1} > t - s_n\\} \\subset \\mathbb{R}^2$. Since $S_n$ and $T_{n+1}$ are independent, this is a straightforward undertaking.
The effect *is* one with probability 1, in your sample. I think what you are probably after is generalizing your sample to a population. Permutation/combination tests may be one way.
Nice job!  I'd like to ask what may be a naive question.  I've seen and used the formula for the standard error of a proportion umpteen times, but I've never encountered the sqrt{1-f} portion of it.  Under what conditions is this needed?
@emanuele http://en.wikipedia.org/wiki/Precision_(statistics)
So the disagreement is he says A & D, and you say A & F, is that right? If you're trying to convince him to give you points for your answer anyway, you could say that surgery E isn't the primary driver of the effect, as I show above. OTOH, F isn't really a good answer, it appeals to a recognition of the phenomenon without a solid understanding of it.  Since there are only 3 choices that advocate for Mercy (the correct hospital) that leaves A & D. Moreover, surgery E *is* part of the effect even if it's not the biggest influence. I would've picked A & D, but the answers were poorly designed.
In the limit $\\sigma^2 \\to 0$, a gaussian distribution $\\mathcal{N}(\\mu, \\sigma^2)$ becomes a Dirac's Delta centered on its mean, $\\mu$. This actually means that it is not a RV anymore: in fact, it becomes a deterministic term, i.e. $\\mu$. This is what the collapse/degeneration actually means: any randomness disappears.
A general way to accelerate an EM is called "Aitken accelerator". If precision is not an issue, maybe try moment estimation or generalized moment estimation instead.
You should explain why this article helps solving the problem. Please provide some details.
The null hypothesis.
consider the answer posted [here](http://stats.stackexchange.com/questions/13086/is-there-a-boxplot-variant-for-poisson-distributed-data/13429#13429). Once the outliers have been flagged, remove them, and use MLE distribution fitting on the remaining observations. It will be more precise for the reasons explained at the link.
And yes, I know that there isn't a correlation, but I'm actually doing this for a podcast I run, "Exposing PseudoAstronomy," and this is the next episode topic.  So I wanted to do the analysis myself and then compare that with what the handful of pseudoscientists claim.  I find going through the exercise oneself is often enlightening, and I'll be in a better position to inform my listeners.
Thanks for your links. Is there also techniques (like time series) who can learn with an learning step :
- 1/A human press a button on the highlight moment of the song.

- 2/ The model find the highlight moments for others songs based on the audio signal.
More generally, assuming constant hazard rates over fixed time intervals (known as a piecewise-exponential model) you can fit fairly flexible survival models in the form of poisson GLMs - if you add interactions between the piecewise constant baseline hazard and covariates, you can estiamte time-varying effects and move away from the proportionality assumption, for example. Sources: Michael Friedman "Piecewise Exponential Models for Survival Data with Covariates", Annals of Statistics
N LAIRD, D OLIVIER "Covariance Analysis of Censored Survival Data Using Log-Linear Analysis Techniques" JASA
500 would be close to the expected income if all other variables were exactly *zero* (not merely constant) --- just like in regression, really.
@Marco The different types of tickets you can earn. So you can earn lottery ticket types 1 through 6, and 42 people won ticket type 1
Update: I am still looking for information on this.  So far my (very amateur) opinion is leaning toward an exponential or weibull distribution, though I am not sure how right these guesses are, or what parameters to use with the distros.  Information on that would be appreciated as well.  Thank you.
I think the OP is asking about a 2-D curve, which is not the same thing as a functional relationship.
It may also be interesting and important to point out that this trick is slick, but will not work in full generality. A nice, moderately challenging exercise is to show that one cannot generate a $\\mathcal U(0,1)$ as a sum of two iid random variables.
I assume you have found the answers yourself in the meantime?
yes, I use it all the time.  If you only have a couple of hyper-parameters (any more and grid search is impractical anyway) then the model selection criterion is likely to be fairly unimodal, so local minima are generally not a problem.
Note that the original matrix does not have to be square but the covariance matrix is necessarily square. In non-square matrices do not have eigenvalues (they do have singular values though).
Do you have any other info on these time wasters? If I was trying to rank these sources of inefficiency and advocate for change, I would go for what's causing the most pain. Not all time lost is the same quality. You might hear from engineers that some source of time loss is very infrequent, but always comes up when they are trying to repair broken servers, for example. Or, 60 minutes lost per week might not be so painful if it comes all at once because it can be planned for, or it might be more painful because it interferes with a release.
This question simplifies greatly when recast in terms of the variances, due to the simple formula for var(A+B).  That version is addressed by the [Cauchy-Schwarz Inequality](http://en.wikipedia.org/wiki/Cauchy%E2%80%93Schwarz_inequality).
Why does it have to be a single model? It makes much more sense to do the classification for gender first and do regression when you're done. If you do it this way you can use any algorithm you want and decouple unrelated issues.
I'm not sure how to answer that. It's a common tool used in model fitting. Here's a paper - www.jrossmacdonald.com/jrmpubs/192LSFITTING.pdf
Yes, you can have any set of constraints you like. You can have b1 < sqrt(variance(x)) and b1 > (b2 * b3). (Not that you'd want to.)
Hmm you are right. Maybe to rephrase my question. What is the difference between g(x) and E[y|x]?
Here are some examples. http://www.google.com/m/url?ei=b548TqDOGtOV9Aa1n_ihAQ&q=http://repub.eur.nl/res/pub/8533/ei2007-07.pdf&ved=0CBkQFjAA&usg=AFQjCNGosUab7wZRy2-bWfqadG4pPFtucQ and http://www.google.com/m/url?ei=b548TqDOGtOV9Aa1n_ihAQ&q=http://ageconsearch.umn.edu/bitstream/49518/2/AAEA%25202009.pdf&ved=0CCIQFjAC&usg=AFQjCNHyALtuXnpPice_ZCaSRTcV1nQK8A the first is a rank ordered logit with random effects and the second is an introduction to the models.
It may well depend on how feature selection is done. But if you think of univariate screening based on correlation tests (with, e.g., Bonferroni or FDR correction) you will not recover variables that are not correlated to Y but have a non-zero partial correlation.
In the meantime I found a way to download the FORTRAN software: you have to register on the page http://metz-roc.uchicago.edu/ and there you find an 32-bit exe-file as well as FORTRAN source code.
Sorry, DA = Discriminant analysis...
In general, mutual information needs some data (often more than other methods). If you see counts 1 or 2 appearing often, it means that mutual information will overfit. The only way to deal with it is to bin some data (e.g. if you have values of one variable `1`,`2`,`3` and `4` then you can use instead `[1,2]` and `[3,4]`).
I would accept both answers if I could as both answer the question comprehensively! Your answer was more immediately useful, while @StasK's solution was more instructive. On that basis I will select his answer as the most relevant to the question. But I appreciate the time and insight that you have both contributed.
The essence of the problem can be appreciated by inspecting the PDF for $((X-\\mu_x)/\\sigma_X,(Y-\\mu_Y)/\\sigma_Y)$. The logarithm of the part that depends on $X$ and $Y$ looks like $-\\frac{1}{2}(X^2 + Y^2 - 2XY\\rho)/(1-\\rho^2)$. Viewing $X$ as a *constant* for the purpose of integrating out $Y$, it is evident that you must compute a Normal integral, which is easy and has an exact solution. What's left depends only on $X$ and $\\rho$: by definition, it's the marginal distribution.
Numerical experiments (a sample of one million iterated $n=5000$ times) and some theoretical work indicate the answers are yes, the distribution stabilizes; but they suggest that no, it is not a Beta distribution. It takes a fairly large sample to distinguish it from a Beta(3/2,3/2) distribution: the stable distribution shifts some of its weight to the left.  It takes a larger sample to distinguish it from *some* Beta distribution such as Beta[1.472, 1.507].
Thanks a lot, this is very helpful!
Generous of you! I've edited my answer to reduce the cross-reference.
By "randomized" do you mean that you simulated scores? If that's the case, and it is not excessively expensive to simulate, you could run a larger simulation (e.g., $n \\geq 10^3$, but this is arbitrary) and use the results to estimate the underlying distribution. Once you have an idea of the distribution, you could then produce a formal test.
Why am I given a downvote?
@Glen_b Clearly even experienced people can draw the line between on- and off-topic in slightly different places. I didn't see there being a statistical issue here, as if there is uncertainty it's about a point in elementary mathematics.
I am thinking in terms of Pearson $\\chi^2$ of $H_0: p_1 = \\ldots p_6 = 1/6$. What significance level should be attached to this test may be open to discussion. So how the "distance" from fair is $z_{1-\\alpha/2}\\sqrt{1/6\\cdot5/6\\cdot1/n}$ where $n$ is the number of times you rolled the die.
Thank you verymuch!
Related: http://stats.stackexchange.com/q/35084/2970
Could you compute hit and false alarms as odd ratios? Then, you could convert OR in d...
@MichaelChernick Note that I put "handle" in quotes. Since this question was originally on SO, I was answering it from an algorithmic perspective, in the sense of the chart from Hastie, Tibsirani, Friedman on pp351 (5th printing) in which they compare various data mining techniques.
Great, thank you. I will go with poorly defined. It is for a paper I am reviewing.
**"In Wikipedia's list of probability distributions, this is but a special case of Beta (rescaled) and Kumaraswamy's (rescaled)."** - isn't this your answer?
Nice answer John.  Why do you think this is not a good idea ?
@Michael Chernick Thanks for the editing !
Choosing a kernel is often a good way of building in expert knowledge of the problem into the classifier, for instance for a handwritten character recognition system, you would probably want to use a kernel that was invariant to small rotations and translations of the image.  Choosing a kernel automatically can be achieved by cross-validation (which is what I would do) or a bound on the leave-one-out error (which can be performed efficiently for SVMs).  However if the dataset is small then choosing a kernel via cross-validation can be tricky due to overfitting the cross-validation error.