tttjjj/elt_bench huggingface dataset:
Hi RockScience, I downloaded the HP-filter add-in (provided by Kurt Annen) and ran the prog. The peculiar thing is that the detrended line looks not much different to the detrended line I got from plotting the residuals of a excel regression of blue line on time. The image of both the detrends is here http://img199.imageshack.us/img199/9916/hpfilter.jpg  The light blue line is what was left after HP-filter. Does this result look sensible? It has a unit root at the 5% level but does not at the 10% level. I will also download R and see if I can get familiar with it. Thanks
To answer the question about whether geographic coordinates can or should be orthogonal just imagine the coordinates as a scatter plot, and what patterns would need to necessary in the point cloud for the two variables to be uncorrelated. A different approach to this problem is to estimate propensity score models, and include the geographic coordinates in the regression allowing for non-linearities. See [Ridgeway and McDonald (2009)](http://dx.doi.org/10.1198/jasa.2009.0034) for an example. [PDF here](http://www.rand.org/content/dam/rand/pubs/reprints/2009/RAND_RP1394.pdf).
Thanks for the suggestion, however, as you hint at, there are problems with categorical variables when the relationship with the outcome is not linear. It only analyzes the 0 and 1 level of the predictor variable, and when there is a different association at different values of predictor, it will not work.
How can we know which screen are you talking about?
@SteveS: It's not ideal, but with `library(mice)` one can get
`df<- boys[,c("hgt", "wgt", "bmi")]`, which has some missing values for the first two columns.
Thankyou! 'group' is a covariate. My question mererly concerns the fact that the long time survivors in the older group have longer survival time than it is possible to have in the later group. What is the effect on the analysis/Hazard ratio. For example a long time survivor in the early group could have 5000 days in survival time while the patients in the later group that also are "cured"(=long time survivors) can only have maximum 1095 days (=3 years). What is the statsitical correct way to deal with this?
Thank you Matthew.
@user55647 - The question was about trends/reversals.   All I can say is, if you're trading some trend/reversal scheme and you can't tell the difference in a so-called "meaningful" trend/reversal versus a random trend/reversal, then the scheme, by definition, is worthless.    The same argument holds for trading schemes based on some other indicator that can't be separated from chance.    A scheme "works" if it has a positive return and can be differentiated from chance.
@whuber, I tried to answer your question with an edit to my own. Basically, my new observations would be meant to test a hypothesis that another variable affects y. I phased this as a new variable *q*, but it could also be one of my *x*s at a values outside the range it's been seen historically. I know that in order to design any experiment that I need to know or assume something about the random variation that will be present. What I'm wondering is how my regression results help me fill in the necessary assumptions. Thanks.
Note that this requires $N$ to be fixed in advance.  Otherwise you start to face the consequences of the central limit theorem
If you mean that your matrix is partly one kind of correlation and partly another, then you are presenting a technique with a messy mixture, confident that the technique will sort it all out. Optimistic to a high degree!
Since the last line of the question implies you can mark or keep track of the buckets individually, why not just put a time stamp on them (stating when each was first known to be filled) and simply empty them in that order?  I don't see how this question involves sampling, weighting, probability, or any other statistical ideas.
You'll better communicate your answer when you use effective formatting. Please read the [Markdown help page](http://stats.stackexchange.com/editing-help).
It might help to explain more about your study design: how are cases and controls sampled?
Great point. I also fell into the trap assuming that 1. is the true setup :(
A general solution is explained in my post at http://stats.stackexchange.com/a/16609. There are, however, multiple different meanings of "weights" in statistics. (For this reason some software such as Stata supports up to *four* different kinds of weight calculations!) In light of this could you clarify what your weights are intended to mean or explain how they arise?
@Nick, thanks for the edit. I think moderator factors cause a constant bias, But this should not imply that measurement error is equivalent to the effect(s) of a moderator(s). Is it right thinking?
In this LR test, I think that the number of degrees of freedom to use is 3 = 4 - 1, since the resticted 'null' model has 1 df while the unrestricted has 4 df, one by region.
Because it is essential to know the rules of the game to answer this question, please describe them or provide a link to a description.
Since Python was discussed enough on CV, I took the liberty to focus this Q on Perl.
Your title seems unrelated to the question text. Please fix that.
Thanks. That sounds about right to me. I'm going to post something about this on the mixed-models list to see if anyone concurs or has something to add. Also, when you have written up stuff using mixed models, what descriptives do you provide?
Moonwalker: We want to rank the users according to who will be the best at predicting the income of future movies.
Hint: When you plot the points $(n, p_n n)$, which lie on the integer lattice in the plane, you obtain a discrete approximation to the line $(n, pn)$. The maximum convergence rate is obtained by choosing lattice points that come as close as possible to the line. Just how close can they come (as a function of $n$)?
The right way to model your data depends crucially on what you want to know, so I guess it depends. As far as running Pearson's correlation (or Spearman's), you should know that the results are strongly dependent on the range of your X variable, whereas the sampling distribution of the slope for a typical regression model remains centered on the true value. Again, which you should do depends, but it's something to keep in mind. Lastly, note that you have measurement error (ie on 0-5%, not 2.217%), this will result in bias, w/ betas closer to 0 that actually true.
glmnet has been undergoing quite a bit of change recently and has had some problems with numerics in the past. Is it possibly due to this? How long since you contacted the authors? I see the current version is 1.7 and was uploaded to CRAN only about a week ago.
This question is incredibly broad but it appears the OP may be asking _what characterizes a distribution that has a closed-form solution for the MLE_ rather than asking for an exhaustive list. In any case, an exhaustive list isn't even possible.
It would be much easier to help if you will describe your variables, show real data or make a script to create a fictional data with given properties.
Please do not cross-post here and on SO. Your question was answered there, so I close it here.
Maybe this will motivate me to download and learn R.  I like this approach.  The ACMVU multiplier that I am using does a pretty good job of accounting for differences in hospitals and the most of them are of similar size.  We began by noting unusually large surpluses at these two hospital systems based on industry benchmarks before we compared them to peer hospitals in the state, so it wasn't a matter of simply picking off the top of the ranking.  Thanks!  You've been a big help!
(Psychometric) questionnaire validation is a complex area, discussed in a large literature and not really a well-defined statistical question that could be addressed by some SPSS procedure. What do you want to do with this “validity”? Where do the questionnaires come from? Standard scales? Previous research? Ad hoc scales you created yourself?
Hi Amit; I made a couple of edits to clarify your post, but I was not sure what to do with "apparel and cloths", which in this context might be "apparel and cloth" (cloth being a mass noun, doesn't gain a plural in this context, in the same way that 'rice' doesn't), or it might perhaps have been intended as "apparel and clothing". I assumed the first, but please alter it if I misunderstood.
For clarification - are the two significant relationships also significant in study t?  If so, given that t appears to be a superset of t+1, why would you pay attention to the t+1 result?
Related thread: [Difference between Cohen's d and Hedges' g for effect size metrics](http://stats.stackexchange.com/q/1850/930).
What happens if $\\mathbb E[Y_n] < \\infty$ and $n \\rightarrow \\infty$?
+1 Similar reasoning provides insight into the notorious [Monty Hall problem](http://en.wikipedia.org/wiki/Monty_Hall_problem).
It's extremely difficult to measure the significance of results that are noted after the fact.  What, precisely, would constitute a "result"?  Having at least one benefit with many more votes than the others?  Having exactly three such benefits?  Having some number of benefits (less than 35) with many more votes than the rest?  Having three benefits with approximately the same numbers of votes?  Etc., etc.  For this reason, you will get the strongest answers by stating your *original* research aim.  That is, *before you saw the data,* how would you have characterized an interesting result?
Just a shameless plug for a new MCMC package in julia: https://github.com/brian-j-smith/Mamba.jl The documentation is very extensive and includes many examples from the BUGS manual.
Included in an edit to the answer just after the place where I said that.
+1, this is a good complement to what's here. It's nice to see that a useful contribution can still be made, even after so many other good ones already exist.
Hi Rob, thanks for getting back to me. I like you idea so I'll give it a try and see if it provides the desired level of filtering. Just one more thing, is there any particular reason for using mean(data) over mean(remainder)? I'm afraid some of my time series might have a somewhat strong trend. STL decomposed series, instead, should not. Also do you think that the approach we outlined so far for assessing forecastability/spotting outliers is good enough to be implemented in a real business enviroment? Or is it too "amateurial"? Would you normally do thing much differently?
No, because it's impossible to characterize a tail that is 40 SDs out just from 1000 iterations.  Picard & Williams explicitly address exactly this issue by showing how you can draw most of the simulated values from relatively far out in the tail.  You won't get out to 40 SDs but you might be able to demonstrate that you are more than, say, six or seven SDs into the tail.  (That's a p-value around $10^{-9}$ or less.)
$s(\\cdot)$ is just your scoring function; $i$ is the rank of the observation in the sample, so that $x_{(1)}$ is the lowest, $x_{(2)}$ the second lowest, & so on, up to $x_{(n)}$, the highest.
The [mixed model tag wiki](http://stats.stackexchange.com/tags/mixed-model/info) provides similar popular references.
That's enough to get me started.
A confidence interval would not even answer the question of what trafic is "normal". You need a prediction interval for that. And the use of percentiles is the most robust way of doing that, especially if you have loads of data.
+1, I always recommend `dlm` and its vignette. The bottom line is that DLM's are much more like programming than most other methods. If you intend to do anything beyond basic modeling and forecasting, you will have to understand the matrices (state space programs in some sense) and methods that `dlm` is generating for you. Most other packages handle the processing of your matrices but expect you to understand how to make them.
Thanks, I am working with hidden Markov models atm so I probably need to read up a bit more on measure theory then. This really answered my questions though so thank you!
Could you please describe the objective of the regression you are trying to create?
If you are not sure, choose R, it's free software. If you will learn how to do analysis with R, there will be no problems switching to STATA.
Using 10 parameters on 28 values to forecast the 29th would seem, on general principles, to be a gross overfitting. What steps have you taken to correct for that possibility?  And how, specifically, does this model address the two sets of questions that have been posed?
+1 -- randomization into the mode is a necessary component of such a comparison. Otherwise, there is no real telling from the observational data and one confounding on top of the other.
Thank you, it seems a bit clearer now.
Thanks Placidia, it makes some sense that the variance is the key factor. But it still seems to me that a statistical significance test takes that into account as well. How is the "distribution of the sample averages" different from "distribution of the individual"? If the individuals have a particular distribution, I imagine the samples of those individuals would too...
You will likely be interested in writing about [*post hoc* power analysis](http://stats.stackexchange.com/q/12118/1036).
Awesome! Thank you so much for writing all this up!
We normally look for more than one-sentence answers. Can you expand on this, such as explaining why? As it stands at present it's more like a comment.
If you want help on machine learning and statistics, you have to provide a better description of the application area. What are you trying to identify? anomalous sequences [are you tracking the path of page views, or just aggregates]? Are you trying to identify bots? give examples of what you consider normal. simple things you could do is simply count number of page views per user. Something more complex would be to build a markov model of p(next page = x| current page = y) ... then you use your markov model to estimate probability of the whole sequence - very low probability-> anomaly
@andrea That makes perfect sense, and would also explain why that number isn't bounded at 1 like I was expecting it to be. If you'd like to repackage your comment as an answer, I'll accept it.
One thing is that in your 2nd equation, your calculation is wrong. $1/(7,7) = 1/(7!/7!) = 1$
I don't see anything in the Box-Cox transformation that would allow it to utilise any given basis set of functions.
"(pixel coordinates of the image columns)" i've heard of this, what software did you use to achieve that?
WOW THANKS A LOT dear Peter :) Yes it is Regression and I would do as you mentioned. :)
thank you! Will do.
Many thanks for this comprehensive answer!
@Emre: Sorry for the long break was stuck in traffic. Ok, this is a small sample set only four, but is should be sufficient to get an idea. http://i.imgur.com/PSpnF.jpg
Any chance you could post a link to some information about Wizard Neural Networks? Google results for "Wizard Neural Network" are mostly for either wizard-style interfaces for making neural networks (i.e., the one question at a time interface with next/back buttons), with one vague comment suggesting that they're just some implementation of a perceptron.
Maybe this is because of the logs in LL that cannot be evaluated at negative numbers. Your starting points may are generating negative numbers.
Welcome to this site! Could you provide more details? As it stands, there is not enough information and the question is not clear. See our Help center on [How to ask a question](https://stats.stackexchange.com/help/how-to-ask).
Please stop changing the title, you are obfuscating my purpose. Perhaps that suggests that this is a poorly defined problem. I'll update it again.
+1. Though, I'd rather say the Pearson coef. "is a measure of strength of linear relationship" instead of `is a measure of the degree of linearity in [= of?] the relationship`
Another reference for the comparison of binomial CIs is Brown LD, Cai TT, & DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101-133. http://projecteuclid.org/euclid.ss/1009213286 (open access). R's `binom` package also has the Agresti-Coull CI.
For any least-squares model and any positive $\\epsilon$, you can add a single extreme observation to reach an R-squared of $1-\\epsilon$. So there is no surprise. But @user603 is right: Why would one want to blindly delete influential observations?
*Simulating* seems to be the wrong word here. *Approximating* seems more appropriate.
@Chi Thanks again! Still, your solution works fine so I am still able to achieve what I want, albeit with a few more lines of code.
Hey, thanks for your comment. Which proposition do you mean, `wilcox.test(DATA$Measurements,DATA$Point)` or `wilcox.test(DATA$Measurements~DATA$Point)` ?
@rolando "Test size" is a standard term: see http://scholar.google.com/scholar?oi=bibs&hl=en&q=%22test+size%22.
Which question? You ask two questions, I have answered the first - "no". Since the second question is predicated on the first being the case (it's not, as I said), the second question cannot make any sense without some new context in which it might apply - but if you must, then I'd have thought that the answer was obvious "no, MLE doesn't in general allow for an f or a t-test because the reasoning on which you based the conclusion was flawed as I said in my first comment". If you edit your question to one that has an answer beyond 'your understanding of MLE is mistaken', I'll do my best.
+1; The sample body is going to be critical.  For example, words like "a", "of", "the" etc may appear a bunch in a corpus and completely bork any attempt to make the distribution uniform.
Interesting question.  Estimating from the sample the distribution of speed and the portion of time vehicles are stationary is trivial, but I can't think of any way to correct for the suspected bias unless there is an external source of information to compare to.
@Corne, I was referring to the "quasi-poisson" model, which is what comes to my mind when I hear about an overdispersed poisson regression model; it's the same as the regular poisson regression model except that the variance is, instead of being contrained to be equal to the mean, is a linear function of the mean.This is in contrast to the negative binomial model, in which the variance is a quadratic function of the mean. See, e.g. [here](http://www.mendeley.com/catalog/quasi-poisson-vs-negative-binomial-regression-we-model-overdispersed-count-data/).
Although bias and variance have potentially different meanings in different contexts, your grasp of them seems flawed, not to say plain wrong.
@glen I updated the question: I'm interested in situations where timing of discontinuities is not known apriori.
It would be really nice if you could add a final comment to your answer stating what exactly the 0.3989 represents to add intuition
How did you assess normality? What are the $x's$ measuring?
That was my core takeaway. The author of the article isn't *wrong*. His "because Math" justification on the other hand, is way off base.
Dear Mr Ming, I am grateful for the trouble taken by you to post a detailed explanation. Am a little confused though. Checked up on some websites regarding Bone Density. The t-values specified in that context are in relation to standard, predefined bone density figures. Comparative t-values in terms of Std Deviations would thus make sense. In my opinion, it may not be proper to adopt a similar analogy in case of paired t-test used to evaluate pre/ post test scores simply because there are no standard/predefined figures to compare with. What do you think? Thanks anyway.
Thanks Rob, that looks like a much better approach.
Try reading http://otexts.com/fpp/9/1
@PeterFlom I used a regularized regression approach in which the variables are automatically standardized.
(+1) thank you very much, very helpful indeed
This looks like it is an assignment and you are required to fill in the dots.  If it is output from a program, it is bizarre. Please provide details.
Yes; this appears to be a question about how to translate English into mathematics :-).  "Drawing" one value $x^{*}_i$ from $(x_1,\\ldots, x_n)$ is just a more descriptive way of asserting the CDF of $x^{*}_i$ is the empirical CDF ($\\hat{F}$) and "with replacement" means all values $x^{*}_j$ are drawn independently from the same CDF.
Thanks for nice explanation. Would you like to share an example preferably analyzed in R.
Can you post some actual output and be more specific as to what your model is? Thanks.
Have you checked for separation, i.e. values of one or more predictors' perfectly separating large from small fishes?
Are you speaking about [Friedman test](http://en.wikipedia.org/wiki/Friedman_test)?
@khan:  why don't you read the wikipedia page and then do some research?
In regards to the table, the stop light color scheme is awful for everything besides stoplights - many people are red-green color blind and yellow rarely prints well. Also you could easily round the decimals in your example to the tenths. Some of the numbers have gone an unfortunate rounding, producing a bit of a lie factor when simply scanning the columns as they are right aligned.
@ rolando - good answer. That being said, missing pairwise approaches tend to confuse the comparison of effects, as they are based on different numbers of observations. Might be something to keep in mind.
Not quite ok for 2. The 'non-linearities' in the predictive probabilities are due to having *unevenly spaced* cut points slicing the probit's (conditionally Normal) latent variable.  Some slices are just bigger than others.  The question for you is whether your data are genuinely ordinal (cut points could be very unevenly spread) or just coarsely measured (effective cut points are 0.5 intervals and do not need to be estimated).  In the latter case, treating the dv as continuous will be ok.  Re 3. the nonlinearity can be dealt with via probit *if you've got it*, but the graph leaves that open.
Statistically, if I choose to use a level with the highest predicted value as the reference, it will mean that all other levels are compared to that reference. Right? In that case, the results interpretation will be based on how other levels are compared or significantly differ (or not) from the reference level. Would that be a good way to go about it?
It's true that hypothesis tests are typically used when people want to show that they are justified in rejecting the null hypothesis. However, that is not always the case (another example might be nested model tests). At any rate, if you plug your data into a normal distribution test, examine the output, & if p>.05 say that your data are normal, then you are following the standard hypothesis testing procedure. Hypothesis testing is really not a good fit for this task, as I state above, but it very much does happen. (IMHO, hypothesis testing is rarely a good approach.)
Why do you see the occurence of a bug as a random thing? If you rerun the code on all computers, will the same computers bug out?
Just a little terminology quibble but what you describe looks like an autoregressive (AR) model to me. It's one form of time series modeling. A time series is simply a series of values measured over time, not a model or something that “works”.
I appreciate your response but what I'm really asking here is if I can claim that the sample is normally distributed?
+1, in particular, I think the part about *standardized* is crucial--otherwise the whole thing is nonsense from the start.
+1 for references, although it doesn't really answer the "why?" question posed by the OP.
Thanks for the reply. So clustering never takes training data? If it does what does the training data contain?
But wasn't the reason for adding the fac fixed effect because the smoothers without it (or with it for that matter) summed to zero?