tttjjj/elt_bench huggingface dataset:
What are trying to do with the data, Robert?  How do you intend to interpret the standard deviation?
Generally, bias isn't diminished by increasing N, eg, the amount of endogeneity will be a function of the degree to which the included variables are correlated w/ omitted variables that have a causal relationship w/ the response.
@nico -- a quick literature search shows that work on this kind of estimation is quite recent, e.g., [Shao and Lii, *Modelling non-homogeneous Poisson processes with almost periodic intensity functions*](http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2010.00758.x/abstract). I don't have access to the paper, but the authors have a [presentation](http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2010.00758.x/abstract) which summarizes their results. No available implementation that I can find... This isn't the way I'd go -- I prefer Bayesian methods. If you like, I'll <cont'd>
The main topic of that paper you linked to is answering the two bullets in your question!  The answers are: 1) in general no, and 2) yes, but it's generally slower than algorithms for the standard lasso.  See another paper by Ryan Tibshirani for an application to "derivative smoothness" regularization: http://arxiv.org/abs/1304.2986
Can you include the code you used to produce this error?
Its nice to read stuff on the internet
I don't see the matrix up there, but that seems like a lot of variables for this little data. I would try looking at the VIFs after a simple regress. Anything above ~10 might be problematic collinearity. Also, you can try tightening up the convergence criterion, or try `ltol(0) tol(1e-7)` to see if the optimizer can work its way out of the bad flat region (in addition to using `difficult`).
For variables that represent total amounts or average amounts it's essentially the same problem as [this one](http://stats.stackexchange.com/questions/59418/interpolation-of-influenza-data-that-conserves-weekly-mean), because any solution amounts to predicting the values for semiannual periods.  Any of the approaches mentioned there (in comments as well as answers) would apply.
This question seems to be perhaps too broad/vague as it stands now, so you might not get too much feedback.  Also, the link you provide doesn't lead to the calculator in question.
The first book seems to be popular from the reviews, will definitely look into it. The second also sounds promising. Thanks.
Interesting question. One bit of advice that comes to mind immediately is to focus on the short-range part of the variogram: that is the crucial part and will do the best job distinguishing pairs of images like these.  (Closely related statistics are [Moran's I](http://en.wikipedia.org/wiki/Moran%27s_I) and [Geary's C](http://en.wikipedia.org/wiki/Geary%27s_C).)  It's hard to give specific advice, though, unless you can more clearly characterize the kinds of images you anticipate processing.
Yes exactly. Often the Ljung-Box Q test is used to test the residuals in order to see whether or not they are random. In case they are then the model is an adequate representation of the data. Something which I forgot to mention in the above answer is also that after you have fitted the model you can look at the ACF and time series plot of the residuals of the fitted series. This will give you a good indication whether or not your residuals are white noise or not (the ACF should not have any significant lags). Anyways, most statistical packages have a command for the Ljung-Box Q test.
Based on this answer, I added a question to the original post!
You'd need to ask the author of that to know their actual intent.  I don't think they mean the word in a statistically technical sense - I think they simply mean that the procedure works as a formal test at large sample sizes rather than being exclusively a small sample test. There's nothing about the way the test is constructed that assumes a particular sample size. I suppose you *could* interpret it to mean that they're claiming it has (no more than) the nominal type I error rate, but I don't think they actually meant that.
Anything would be appreciated, it's all a learning experience :)  The exponential for time makes sense for the time parameter itself, but how do you combine it with the other parameter(s)?  Simple multiplication itself doesn't quite give the correct results...
**Example 5.5** (*Unconstrained geometric program*) on page 254 of B&V will get you very close. With your stated background, I suspect you can take it from there. :-)
No, I don't know much about maximum likelihood estimators. But please feel free to answer with the assumption that I do know everything about it, as most people reading this question in the future will probably have a working knowledge of this.
Really appreciate it !!!
Re bullet 1: This topic has come up elsewhere, but I think it's important. Correlations among the independent variables tell you little about which ones to use for regression with another ("dependent") variable. It is possible for two IVs to be strongly correlated (with correlation $\\rho$) and for one of them to have no correlation with the DV and yet the other can be substantially correlated with the DV (with correlation $\\tau$). The only restriction is that $\\rho^2+\\tau^2\\le 1$. If you don't take care, you could throw away the only IV of any value in the regression!
Hello, thanks a lot for your comment; the approach in the NIPS paper also may be computationally complex, since it involves a matrix inversion operation (which is $\\mathcal{O}(N^{3})$). The choice of a kernel-based method was driven by the fact that such type of methods allow you to work on structured representations (e.g. trees, graphs, documents) but indeed also the Euclidean distance can be "kernelized".
"My goal is to use this data to provide a probabilistic guess of where a certain name is from."  You already know which country each name is from.  My sense is that, given a new name, you want to guess which country that name is from.  You could get into all sorts of complex methods such as neural networks, but why not simply assign as your guess the country that has the highest % of its names matching that name?
Note $X^\\mathrm{T}$ means the transpose of the matrix $X$, but $XT$ means the product of matrices $X$ & $T$. (I've fixed it.) Also, though the equation's fairly standard, it would be helpful to define the symbols.
Are you asking about how to do this in SPSS? If so, the question is off-topic here. Or are you asking about what statistics you should use? If so, please clarify what your data are and what your hypothesis is. "Employees are satisfied" is not a testable hypothesis.
A definition, references and links can be found here - http://en.wikipedia.org/wiki/Simultaneous_equations_model
If the model is "true," the statements are equivalent: $\\varepsilon_i\\sim N(0,\\sigma^2)\\Longleftrightarrow y_i\\sim N(x_i^T\\beta,\\sigma^2)$, since $x_i^T\\beta=\\mathbb{E}[y_i|x_i]$. I'm not sure if I got that precisely right, but that's the idea at least. This is why people use the term "variance" so loosely when they talk about regression, because it doesn't matter which one they're referring to.
Agreed with Glen_b. Your question is confusing. Saying what you did is good, but more importantly, tell us what you are trying to find. For example, are you interested in changes across time (across measurements within the individual)? Why did you take multiple measurements? How did you determine the timing of the measurements?
My question: if I define $D'=|D|$ (or, in other words, we always test the model with the lower likelihood against the model with the higher likelihood), wouldn't $D'$ have a $\\chi^2$ distribution?
One level is the reference level and it is "hidden" in the intercept.
Thanks for the response! I see that adding a low-variance noise to high-variance signal can skew PCA quite a bit. However, my focus is on showing something similar for noise in graph Laplacians -- it seems a little bit more difficult for that case; however, in that case, I suppose playing with the eigengap can yield something. Thanks again.
Inspite of the fact that you are mining for URLs I don't think this question fits this site.
$\\bar{x}$ and $\\bar{y}$ may also change between the two samples.
Thank you! You deserve the bounty )))
First, $\\sum_{n=1}^\\infty \\mathbb P(|X_n| > n) < \\infty$ imples $\\mathbb P(|X_n| > n \\,\\mathrm{i.o.}) = 0$. The "infinitely often" ($\\mathrm{i.o.}$) part is *very* important. This now comes down to understanding the definitions of (i) "infinitely often", (ii) "almost surely" and (iii) $\\limsup$.
$F(L) < 0.4$ means that c has <40% chance to be below L and >60 to be above. Express what does it mean for the expected profit of the company, the environmental damage.
+1 very nice argument from looking at a scatter plot of the data.  You remind us of the importance of scatter plots and graphics in general that should be the first step in the analysis process before any models are fit to the data!
Probability of observing value above median is one half, but your quantity is log-normally distributed for large n, which has different mean and median, so you shouldn't expect that probability to approach 1/2.
Thanks Peter. But, in the final model, should I discuss the beta weight of science, which is SS but was controlled in previous models? If I do have to discuss, what is the point of doing a hierarchical regression but not simple multiple regression with all bundled in one equation?
What exactly does your notation "$\\mathcal{N}(x_i;x_{i-1},1)$" mean? Specifying a univariate Normal distribution requires only two parameters but here you have three. Also, what does "probability...at point $x$" mean? Since for $i\\gt 0$ the chance that $x_i = x$ is zero, almost surely no value in any given realization will ever equal $x$ and so it is not apparent how $S$ would even be applied. Please enlighten us by means of suitable edits to this question.
Well, fixed effects effect the mean of a joint distribution and random effects effect the variance and association structure. What exactly do you mean by the "mathematical difference"? Are you asking how the the likelihood changes? Can you be more specific?
@MatteoFasiolo For each parameter use one vertical plot. Put the different model results (ie each method used) next to each other in the same plot and colored different. Perhaps leave a blank space in between. This should be 4 plots with 30 lines each, which could possibly fit on one page. Or perhaps I do not understand your data.
@whuber :I agree the example is a bit contrived - my intent is to gauge potential impacts for certain variables in a feasibility study and I wish to be able to simulate it to see how much it would vary and get some more insight. Could you suggest a better example? I'll add it to the question to make it more clear!
Check the Mathematics site for an answer.
I assume you'll aggregate the nursing information and add it as a hospital characteristic to the model of patient level outcomes.
Frequentist tests don't actually allow you to assign a probability to the truth of either hypothesis, so it is quite difficult to put it into words, but essentially yes, if the null hypothesis is not rejected and the test has high statistical power, then it would be reasonable to conclude that there was no difference in the means.  However, computing the statistical power is not always straightforward, in this case I suspect it would depend on the minimum magnitude of the difference that was meaningful.  A Bayesian approach might be easier, using Bayes factors.
Thanks, it's intuitive that objective function decreases, but I wasn't sure that it decreases strictly. I wanted to make sure that there is no patological case just like in linear programming
+1 for managing to steer the dinner table conversation from Gangnam to statistics. We need people like you!
There is a concept of an adaptive bandwidth for kernel density estimation, and the opposite would be a fixed bandwidth. Adaptive encompasses a specified number of points (no matter how far away) to estimate the density, whereas a *fixed* bandwidth only calculates the density based on a specified distance away from the point. All of these might suggest in this context **fixed** is the opposite of adaptive, although there are likely different examples not in the discussion where this does not fit.
Multi-level modelling is definitely easier for bayesian, especially conceptually.
1. All distributions used are location scale invariant.
@ttnphns No. **Mean** absolute deviation error.
I enjoyed the $P(L>t)=P$ *(no hits in time t)* explanation. This made sense for me.
It looks to me like it might make sense to model two things - 1: the trend in the *probability* that there's at least one "frost day", and 2: *Given* there's at least one frost day, consider the trend in date.
Thanks @NickStauner for pointing me to additional useful packages.
@dr It should be identical. The point of this derivation was to *justify* this approach using basic principles of statistical inference and psychology, because it appears that the crux of the issue concerns the right way to handle the ties.
@gmacfarlane the correlation is the correlation between sales and profit for each month
Thank you! Very informative and lots to think about!
The features are real numbers which I will create by feature engineering.
That's what I'd point to you. Good luck.
If you can clarify your understanding of your question further, you may get some interesting answers. Are you interested in probabilistic bounds, for example (e.g. ones which are very likely to be sufficient, but which will fail occasionally and need to be recalculated)?
@whuber -- OK, but the significance test for the interaction of time and probability is reported in the sentence preceding the one I highlighted. That's not the one I'm asking about: I'm trying to figure out what term from what model gave an F value of 171.38 on (1, 116) degrees of freedom, as reported in the highlighted line. It can't be the probability by time interaction, because that has already been reported on the previous line!
Ok, but an equal tailed interval and HDI interval does corresponds to minimized expected L1 and L0 loss in the whole set? So what I'm looking for is the interval that mimimizes the expected L2 loss? Still I'm wondering if it has a name, how it is calculated, and why it isn't more widely used...
... and then, of course, we conclude $|\\rho| \\le 1$.  This illustrates the answer provided by @vqv.  @ronaf: why didn't you upvote @vqv's answer then??
@NRH I've worked through the MGF myself in the symmetric case ($\\lambda_j = \\sigma^2$) where $p=2$ and instead of $b_j^2 \\lambda_j$ in the summation, I have $\\mu_j^2$. Simulation verifies the first moment. It's possible that this is the "linear function" you mention and that this is peculiar to the symmetric case, but I thought I'd point it out in case there's an error.
Thanks for the feedback-I've added some longer term numbers to explore the seasonality angle.  I was concerned about using a prior year's period because of the impact of long-term trends, but you covered that in #2.
context might help. Where do these phrases come from, and what's he context in which they occur?
Cross-posted: http://math.stackexchange.com/questions/766939/what-is-rm-cove-i-hat-y-i-in-simple-linear-regression
Also, I just added a note that the variables are actually not independent
How were the categories created? For instance, the "obvious" answer that you can use t-tests (which is wrong due to the multiple comparisons) and the better answer that you can use ANOVA are both incorrect when, say, the categories are the quartiles of the $16$ values. It would also help if you could tell us the *purpose* of the comparison.
@Whuber. I do get what expectation and probability is but my question is how come p(x|D) is expectation in the above case it is probability isn't it?
A similar question has been asked here: http://quant.stackexchange.com/questions/1480/what-is-the-ideal-ratio-of-in-sample-length-to-out-of-sample-length The consensus there seems to be that there's no clear consensus regarding the IS/OS ratio.
I have seen this paper - very interesting. Two questions. 1) Lets take logistic regression. It sounds like the only way to conduct CI or hypothesis tests is to build a model in the style of hosmer and lemeshow (precluding any data sets with big p)? So you are left with "using" the model for only point estimates? 2) Your paper discusses the lasso among other alternatives. Are you of the mind that this allows later hypothesis testing or is "simply" given as a better option of model selection?
If you're using a 2SD criterion to remove values from 10,000, you will likely be removing *hundreds* of them, not just two.  Such a criterion is a poor one for identifying genuine outliers.
@shabbychef It's not so obvious to me.  You can have the expectation grow exponentially (or even faster) while the probability of dying out still approaches unity.  (For example, consider a stochastic process in which the population either quadruples in each generation or dies out entirely, each with equal chances.  The expectation at generation n is 2^n but the probability of extinction is 1.)  Thus there is no inherent contradiction; your argument needs something additional.
This question is too vague. What kind of confidence intervals? What sort of models are you interested on? Consider taking a look at the literature before posting 3 questions in a row.
If this is for a subject, please add the homework tag.
@jerin "Please make this data set stationary" does not appear to be a legitimate question for CV; that's a request for someone to do your work for you. Please see the [help](http://stats.stackexchange.com/help) which includes discussion of whats on topic. Further, use of extreme begging is both pointless (since everyone who asks a question can be presumed to want their questions answered), and carries the dubious implication that somehow you think people are withholding answers they would otherwise give, if only people with questions would plead more.
(+1) this is a very interesting problem. I am curious: Which domain ? Can you tell us some more about the actors and objects (what are they, robots picking cans) ? It may not matter for the question itself, but as I said, I am very curious ;).
+1. It sounds like your "negative probabilities" are just signed measures, right?
OPTICS seems quite interesting. I completely agree with the multiple runs, but want to annotate the '1000 other clustering algorithms'. The number is probably much larger still, but the fact is, only a small percentage are available as software. I think it is a reasonable litmus test to see whether anyone (especially the authors) made available a usable (not necessarily scalable) implementation. If this is not the case, the algorithm may be removed from consideration.
@hhh dividing the covariance like this is incorrect.  Think in terms of scatterplots: behind each year's paired data is a set of about 250 pairs of daily returns.  The question is trying to recover the scatter of those daily returns from observing the scatter of these averaged-out points.  There is no fixed relationship, or even useful bounds available, unless one makes strong assumptions.  In particular, dividing by two--even assuming no temporal correlation--will not generally produce the correct answer.
+1 for the Emacs point - I should take a look at that.
It seems to me that imputation of missing data only makes sense when framed into a specific statistical context (regression, classification, scores reporting, etc.). Could you specify what you have in mind here?
You are going to need to get that information somewhere, or you are just stuck. Some distributions have SDs / variances that are functions of the mean, so if you knew if it were one of them (eg, Poisson), then you could figure it out. But basically you are out of luck.
1. Do you have a sample set of data $\\{ X_i \\}$?  2. When you say "the improper prior is used", which one?  Perhaps the one with density proportional to $1/\\theta$? Or perhaps something else? 3. Which *credible interval* (not *confidence interval*) do you want?  [Wikipedia offers three](http://en.wikipedia.org/wiki/Credible_interval), though I don't think the third is particularly useful.
I want to see if the travel pattern in the town x is different form travel pattern of townswith simillar population. for instance if the share of the travel mode "car as driver" is considerabely higher or lower than other cities. the numbers showed aboce are represtative for: car as drive/car as passenger/bus/bicycle/by foot .
[here](http://faculty.chicagobooth.edu/nicholas.polson/research/papers/Horse.pdf) is a link which compares the shrinkage properties of the lasso and horseshoe.  Similar results would occur for the generalised double pareto, as it has similar tail behaviour.  It has a lot to do with local-global shrinkage rules.  Lasso is just a global shrinkage rule, so it has real difficulty when you have both sparsity and large signals.  I'm pretty sure this is known via the inconsistent parameter estimates for non-zero betas.
You might want to clarify what your data are and what your question is. I suspect that you want to know if there are main effects of $X$ and $Y$ and an interaction between the two, in which case you might want to run `anova(lm(..))`.
It seems to me that this question is about `clustered sample` and not about `clustering`. Please check you tags.
In any case, an important thing to keep in mind is that the relevant analysis is not a simple function of the level of measurement of your variables. The most important question is “What do you hope to learn from this experiment?”
You must reformulate your question, as it stands, it does not make very much sense (maybe only a TeXnical problem?). As it stands, $f(x)$ does not depend on $\\theta$, so your parameter $\\theta$ is not even identifiable! (If you try to normalize your density so that it always integrates to one, you will cancel $\\theta$!!!)
You seem to miss the point. There must be more x-values than the three you drew (the line you drew is not remotely near the least squares line for those three points), and so we need to know where the mean of all the x-values is, as whuber already said. Without that information, there's no way to say which has the higher leverage.
The big issue with "day of year" concerns how to encode it. Ordinarily it would be represented as a Julian day between $1$ and $365$ or as a decimal year from $0$ to $1$, but neither of these is appropriate because this is a *circular* variable: the Julian day of $1$ immediately follows day $365$, for instance. Thus, in particular, "upper" and "lower" limits are meaningless. (There is also a minor issue of how to handle leap years; this could be resolved in various simple ways.) The other big issue concerns handling years where freezing does not occur: this is *not* missing or censored data.
@Jonathan I don't have a good sense of how much noise is required before random forests start outcompeting the various boosting methods, but my impression is that it's quire a lot.  The `gbm` package in `R` has a functioning "weights" term, and you can increase the randomness in a very similar fashion to random forests by decreasing with the "bag.fraction" term.  It seems like it's worth a try.
@davips: If the models are unstable, the optimization will not work (instability causes additional variance). Choosing the one model with median (or average) performance will not help, though. Instead, if the models are unstable I'd recommend either to go for more restrictive models (e.g. stronger regularization) or to build a model ensemble (which is fundametally different from selecting one model).
Have you looked at the p-value (0.09387) reported at the very end of the quotation?
It works for me on R 2.12.1.  The problem seems to be specific to R 2.13.1 and 2.13.2.