tttjjj/elt_bench huggingface dataset:
Extrapolating is like the technique used to estimate the mean and covariance matrix for unequal data. Basically, you would estimate the mean and covariance of the data that is longest and then regress the smaller dataset on the longer dataset. The error estimates would be calculated over the shorter period, but you could use the whole series of the longer dataset to predict the values of the shorter series at the longer lengths.
Are you referring to a deterministic relationship?
How is it that you have a 3-level categorical variable, but only 1 coeff? I suspect you haven't provided enough information for this question to be answerable.
+1 The SVM is an approximate implementation of a generalisation bound that is independent of the dimensionality of the feature space, so there is good reason to suppose it will work well with redundant features.  Selecting the features using e.g. cross-validation invalidates that bound, and is very likely to result in over-fitting the CV error, so quite often feature selection will make the model worse rather than better (although of course the CV error that you have optimised will not reflect that).
Maybe you will find this link helpful: http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysi
Aww. I thought this was going to be a question about propagating loopy beliefs.
Your opinion's one supporting argument sounds an awful lot like the logic behind the model that you criticize for that very same logic. If the (generational?) environment isn't similar enough now to when the model fit Myspace, why is it similar enough to base any opinion on Myspace's history? Furthermore, is Facebook really just another social network that will behave like every other? It's different enough for doubt in plenty of ways, as are the times, so again, I don't see how it's safe to assume its chance of a similar fate is big **within such a short time frame**.
@Zach: you and me both, it would make a lot of sense... But the paper behind my third link notes that simulations were done in R using `lars` and `glmnet`, so it might be worthwhile to mail the authors and ask for their code.
Could you elaborate please ? Do you mean you have a set of variables, a linear combination of some of these variables, but you don't know which ones ?
The usual example of two "uncorrelated but dependent" rv's is to consider the square of a Uniform rv symmetric around zero.Here too, we have the square of a standard normal (a chi-square rv that is) that is also uncorrelated but dependent with its parent standard normal. This should permit us to understand a general rule (not unique of course) as to when we can have this phenomenon of purely non-linear dependence.
@FrankHarrell: Isn't Predictive Analytics only a smaller subset of Statistics? Those two terms don't seem identical to me.
I've taken the liberty of adding a display of your data to your question, which highlights (1) the discreteness issue that I mentioned; (2) the great range in size of values across both of the data sets; and (3) that the distributions are almost identical. There are many issues that should be addressed here (see my comments above) before much can be said, but I think the main problem is as addressed by gung and the link whuber has indicated. I also notice that there's another big spike at 1.98, which I missed in a visual inspection before.
Welcome to the site
ROCR is not bound to a specific model. The only information you need are the true labels and the predictions of a model. Can you please provide some code and tell us where exactly you get stuck ?
I'm rather convinced by your argument that R will gain more SPSS converts if they are provided with easy means of achieving familiar analyses, even if these analyses require more nuanced consideration than is typically applied. Then, once we have them hooked on R, we can attempt to convince them that the familiar approach isn't necessarily always the appropriate approach. I think I'll still put the default as type=2 with a big warning in the documentation that this default may yield results that differ from SPSS and links do documentation on the type II/III distinction.
"Something that arises in nature" and "something that wikipedia has in its list" are likely to be very close to disjoint sets.
Le Max, please check that the edits people have made to improve your question leave it asking what you want.
@whuber ... or I was channeling James Joyce when I wrote it.  I'll redo the post later today (it's midday my time.)  BTW +1 for the tact!
In fact, it's used in one exercise in a lecture I attended.
(For those who may not remember, in spite of the Literary Digest predicting a landslide for Landon, Roosevelt was the one with the landslide, and Landon picked up only two states with 8 electoral votes.  There has been some disagreement as to the particulars, but there's a general agreement that the Literary Digest poll was seriously skewed due to poor sampling -- they polled their own readers plus some addresses gleaned from telephone books and auto registrations.  This resulted in a very lopsided sample, skewed towards more wealthy citizens.)
Btw, I just did this today!!!
[This Wikipedia article](http://en.wikipedia.org/wiki/Student%27s_t-test) plus the help page for R's t-distribution functions (got by `?pt`) -- see especially `pt()` -- do have all the info you'd need to do this yourself. And you'll learn a *lot* about stats and R if you do that.
As an alternative to Laplace smoothing, would you consider a Bayesian approach, with a choice of prior that puts some non-null probability on all transitions?
Thanks, that link is quite helpful
Sounds like you are reinventing Cohen's Kappa.
I agree. Standard deviation is the *right* way to measure dispersion if you assume normal distribution. And a lot of distributions and real data are an approximately normal.
Could you clarify what the numerator and denominator are in your response variable?
This question appears to be off-topic because it is about whether this feature exists in R.
Good question. I've used a version of this in class (trees in a park). They get the idea of sampling, but tend to miss the need for operational definition: when do you start calling it a tree?
Weren't these questions thoroughly answered at http://stats.stackexchange.com/questions/16921/how-to-understand-degrees-of-freedom?
I guess I should add based on the other comments that if any step of this doesn't make sense to you then you're trying to go too far too fast. You had to learn to add before you could multiply before you could do algebra before you could do trigonometry. The same thing applies here.
Could this relate to Copula? It is uncorrelated distributions, but with dependent structures. I am not sure whether the question is correct, because statistically independent means uncorrelated, but uncorrelation doesn't mean independent. The question maybe another way around.
@Anh: both of the splits can be done by resampling (e.g. repeated cross validation) instead of setting aside only one small data set.
@onestop Yes they are the only available in my library.
I don't understand your second and third questions.
Thank you very much to everybody!
May be I don't completely understand what you asked, but if you do not want to use auto.arima and want to be able to manually play around with the AR, I and MA terms and do it for each country, I don't really see a way to fit it in for loop..moreover, auto.arima does that for you anyway by checking AIC levels and selecting the best model of all..
I might add that VARs are for multivariate time series data, and that each outcome is modeled using previous values of itself as well as other lagged outcome variables (that's the autoregressive part).
@John Originally the color was added just to aid the eye. But having looked at the result, I find it interesting that in five of the six years preceding the Internet stock blowup in 2000, the orange weeks (roughly late summer) all exhibited strong upward trends. Afterwards, that tendency appears to have vanished.
How do you imagine cluster->PCA->plot path to yourself?
Fuzzy splits, got it, in the spirit of extreme RF (but possibly not so extreme?). I will try this as your explanation makes sense to me. Thanks.
As Rob Hyndman suggests (and really, he's the expert), backcast is the most common term.
The problem might be in your conflict between "a single temperature record for the whole region" and any interest you have in intra-region variation.  A solution might involve some way of reconciling these two issues eg partitioning variance into intra- and inter- region components.
No problem. You may already know this, but `?command` will print the help for a command.
[Searches](stats.stackexchange.com/search?q=[time-series]+trend+seasonal+holiday) can often turn up useful posts. I see a couple there that seem to be relevant.
Hey, there are two parts to the questionnaire, 50% were given part a first and 50% part B first. Yes people have a score, for every correct answer they get a mark and this is out of 25.
Nice answer (+1) Indeed, you do not need to worry too much at that point. You need to get _some_ data to know something about the variance, which in turn will tell you how much data you need.
I downvoted this question because anyone can easily and quickly resolve the distinction between [covariate](https://www.google.com/search?q=covariate+definition) and [covariance](https://www.google.com/search?q=covariance+definition) by consulting a dictionary.  We request that you [do some research into the answer](http://stats.stackexchange.com/help/how-to-ask) before you post any question, because "sharing your research helps everyone. ... This demonstrates that you’ve taken the time to try to help yourself [and] it saves us from reiterating obvious answers."
Coding matrix = basis matrix = design matrix shows the _values_ of the set of conrast variables, how they correspond to the categories. Contrast coefficients matrix = L matrix helps to _interpret_ what parameters in the model will mean (represent) statistically.
That thing you say that you know ($p = N_1/N$) seems to actually be an approximation, and in this situation, I don't think it's a good one. Have you seen the [original paper](http://www.ling.upenn.edu/courses/cogs502/GoodTuring1953.pdf)? Even if you used smoothing you appear to be well outside the level of data that the technique would be used for as is. The ideas can probably be adapted to this situation but the uncertainties would be very large indeed.
Of course. How would you for instance find out that a variable is not 'statistically significant' if not by regression?
for measuring the overfitting, you just need to nest the cross-validation.  The outer cross-validation is used for performance assessment, and withing each fold of the outer cross-validation an "inner" cross-validation is used for feature selection and model selection etc.  That will give you an unbiased performance estimate.
@StasK Also, I still hope there are other people who have similar problems with these concepts as I do and not all of them might have access to the full text of this Springer Link.
Corollary: the absence of an arrow between $A$ and $B$ implies that $A$ 'does not directly cause' $B$. (Of course, there may still be indirect causal effects of $A$ on $B$).
Regarding logit regression, this will not do what you want as it will take adjacent datapoints information, and many True samples next to a few false will change the prediction of the false.
Regression aims at *identifying a function* to describe the expected value of $Y$ (the dependent variable) given $X$ (the independent variables). Interpolation uses regression for *predicting* the value of $Y$ at given values of $X$. The difference is subtle but comes to the fore in models where the $Y$'s are correlated, because then the predicted values typically *differ* from their regression values. Neither regression nor prediction apply directly to interpolating in mathematical tables, which usually are assumed to have no random error, but their *algorithms* can still be used.
I'm not sure how to interpret the SAS probabilities... is that 0.0072%, 1.884%, 0.954%?  These seem terribly small for a dataset with 36% of the observations (adjusted by weighting) = 1...
(Disclaimer: I don't know Ruby, so you may have to adapt the idea to have the right syntax. The `log` in my "code" is the natural logarithm.)
Glen_b... thanks!
@TheBug, no, the SVM itself is still trained using SMO, but the hyper-parameters ($\\nu$ and any kernel parameters) are optimised by gradient descent (or Nelder Mead simplex) minimisation of some suitable model selection criterion (e.g. cross-validation error).  At each step you have to train several SVMs with fixed $\\nu$ in order to define the simplex or to compute the gradient (using finite differences if it can't be done analytically).  The cross-validation error is not generally unimodal, but it is normally smooth and such methods work well in practice.
Thank you for this thorough list of suggestions. I have looked into a lot of these factors before, they are all good points. But none of these points speaks to my question about regression to the mean effects, sometimes called a "regression artifact".
I just saw test inversion's addressed in DavidR's answer to that question, in any case. @user3269: perhaps you could have a look & see if you still have any doubts.
(+1) A very precise and detailed response.
This tip helps me get closer to my goal of being able to compare the relative importance of my predictors - at least now I have standardized beta coefficients.  Still, would be nice to find some package or process like R's 'relimpo' package that will work on a NB model...
If the variables are scaled differently, then not necessarily. If they've been standardized first, then sort of; it is at least the one that the regression picked out as seeming to be the most correlated with the response.
It might be a multi-pronged approach depending on the sparsity of the data. If you have "enough" specific matches of people who wear Nike size 9 and Adidas size x, then you might use the average of the specific matches. Then, if there aren't enough, you use a general sizing rule like +0.5.
@Dmitrij Please put your comment as an answer, this might help the OP to refine his question or engage into a more constructive dialogue. (As you said this has become a long-lasting series of questions now). @Imelza You already asked 5 questions, and got some comments on them (see e.g., [here](http://stats.stackexchange.com/questions/7238/matlab-interpretation-of-xcorr2-cross-correlation-function) and [here](http://stats.stackexchange.com/questions/8116/cross-correlation-in-matlab)) but you didn't address any of them. Any reason why?
I am also working with correalted data and i am getting quite good results with lasso.
Sure, if you can re-word that for a 7 year old I'll register 10 accounts and they can all give you +1 :). Less factiously, I was trying to circumvent that by answering in such as way as to not require understand of what a function is at all. You always lose something when you simplify.
Is this a question that pertains to a class you are taking, or a subject you are studying? If so, please add the `[self-study]` tag & read our [help page](http://stats.stackexchange.com/help) on such questions.
plm as far as I know does not have functions for fitting discrete data models. `lme4` is a good suggestion, but should be used with care, since it assumes that all the independent variables are exogenous, which is not always the case for panel data in econometrics.
Oh i see. Yeah I could see why that axis label might be confusing. Technically, the X-axis is not "Ball Color" but rather the value of the data point associated with a colored ball. Remember that each ball has a color and data point assigned to it. The color of the ball corresponds to which component distribution its data point was drawn from.
@Charlie thanks, I found "an answer" that will have to do for now, see below
Do you really want to test *significance*? That is as much a measure of how much data you have as it is a characterization of the data. The questions of *scientific* interest would focus on effect sizes (that is, the coefficients) and goodness of fit diagnostics (such as tests for interactions and nonlinear relations).  BTW, welcome to our site! It's nice to see such a thoughtful, important, well-expressed question.
Would some GLM with a sufficiently right-skew distribution - perhaps gamma or inverse-Gaussian - solve this problem?
Do you think the procedure that I wrote the code is right ? because I am hesitating if pca distance-distance representation is useful for me.
@NickCox +1 -- indeed the inverse of any continuous-strictly-monotonic-cdf-on-$(-\\infty,\\infty)$ maps $(0,1)$ to $(-\\infty,\\infty)$. There's uncountable infinities of choices.
@James, this again depends. But instead of using only every n$^{th}$ point, I'd recommend averaging/binning every n points, so you reduce the dimensionality and increase the signal to noise ratio. Finding out what (whatever your data point dimension is; I'd put "spectral" here) resolution you need is IMHO one point of the characterization of your problem/data/classifier characterization.
I am using accuracy = TP/(TP+TN)
1) What do you mean by "p-value" and "whether it's just noise"?  If it's random, and you have no information about why you see e.g. a 1 vs an 0.5 on some observations, it is just noise, or it may as well be.  2) With the sample size you have, that bump around 0.6 is likely to cause you big problems fitting a simple, off-the-shelf distribution.
+1.  The idea that the truncated normal will result in a different posterior is interesting and surprising! I did comment that I was skeptical, but I deleted that comment. I'll need to think of this a little more.  Normally, I find the Likelihood Principle to be 'obviously true', so I should think about this a bit more.
Define "analysis". Do you want to compute means and medians? Do you want to find patterns and relations? Do you want to build classifiers? Besides, Hadoop is not a data analysis tool per se; it's an OS/file system/job scheduling layer that allows other programs to run over the distributed system.
Also, in that it is a restated question, are you suggesting to include comments about why $d_2$ is the best measure of distance between functions?
Bad idea. At the least, after eliminating some variables, re-fit the model: the coefficients will change and the model will improve (or not get any worse). Better, review the threads on this site about [tag:model-selection] and search for references to "regularization."
Bought the book @Hogan but it doesn't include the part about other companies being mixed in though.
One thing that does have a name is ["Lie Factor"](http://www.infovis-wiki.net/index.php/Lie_Factor) (Tufte).  The Lie Factor on the x-axis is 10:1, owing to the nearly hidden, nonsmooth jump in scale at 1 micrometer.
An anonymous user posted the following comment (as an attempted edit): "`pchisq` only calculates the cumulative probability ($P(X\\le x)$) whereas p value is $P(X\\ge x)$"; `pvalue=1-pchisq(chisq,df)`; & "The result when you calculate the p-value manually is different from the p-value that the goodfit function supplies. I dont know why this is so."
I am trying to get measure trend and seasonality (in case they exists), in order to get the inputs for later estimating weather risks. Regarding the seasonality I have been using monthly dummies and adding these monthly dummies into the OLS model (being careful with collinearity, of course)
+1 For bringing attention to this book.
Amazon has been one of the leaders in the (internet) technology industry in experimental design for online advertising and website usage. You can be sure that they actually *are* quite sophisticated in their statistical approaches. :-) Your point is a good one. To take it a small step farther, can you imagine if Amazon were doing something "more sophisticated" and someone checked them by using a simple average, found that some items were ranked "higher" than their average and others "lower", raising a fuss and leaving Amazon to try to explain their "hidden biases" regarding products?
A confusing aspect of this question is that it refers to p-values for the coefficients and to residuals as if they were somehow the same thing, even though those are entirely different. Could you clarify?  BTW, there's nothing unusual with p-values of coefficients being this small (or even much, much smaller), especially with large datasets.
I'm afraid this question is stll very unclear.
Community wiki...
I've added some fabricated pictures (RGB, equal size and resolution).
Ah, yes. Typo got me. :)
Fred: Specify the doubly integrated series SSt= St*{[1-B][1-B**12]}**-1 and similarly create PPt from Pt. AS you said the coefficients will capture the instantaneous effect and the gradual decay effect
Could you explain what you mean by successes being "in a specific order"?  On the face of it, outcomes are either successes or not.  All successes are treated interchangeably and likewise all non-successes.  Therefore there is no sense of "order" for either type.
Thanks for the recommendation. I found a document of [SAS](http://support.sas.com/resources/papers/proceedings11/335-2011.pdf) and [R](http://cran.r-project.org/web/packages/betareg/vignettes/betareg.pdf) about this technique and beta regression seems very appropriate.
And how many dimensions is it? Is it just univariate?
There is no open source solution that I am aware of. AUTOBOX does market a Linux Version but it is far from free. At least now you have a clue as to how to proceed with your analysis.
Check out the comments and links available in this thread: http://r.789695.n4.nabble.com/allocMatrix-limits-td864864.html
Good info. But you may want to update that link to Wicklin's book =) For example, to: http://support.sas.com/publishing/authors/wicklin.html
Thanks @Mike_Nute.
your experience -- has run on 1k / 100k points, in 1d / 2d / 3d -- would also be useful.
Actually for my case it is sort of similar. Because FP and FN cases are gonna be costly in my model. I eventually ended up doing something similar to what you suggested "using multiple measures". I calculated the F-score for each class label, and to assess the models I use both these values along with some cost function that uses precision (for both classes) to calculate profit and subtracts from it the loss incurred from FP and FN cases.
@Robbie - sounds like you are looking at the F test that tests simultaneously all the variables in your model against a null model which has only an intercept.  So "yes", that test just shows that one of x or z (or the two in combination) is associated with y.  It is possible to construct F tests to test any combination of variables however - see http://stats.stackexchange.com/questions/21040/collapsing-regression-of-nominal-data-to-single-significance-value/21353#21353.
Thanks, my adviser is looking for positive correlations, not me. My intention for the data was to show correlations between spending on certain items (mobile phones, clothe and shoes to peer influence, advertisements and saving-face). I noticed that my subjects were not fully truthful in answering my questions, and knew this would affect any information I was looking for. Although, the correlations are weak and negative, my adviser wants positive correlations. I'm pretty much stuck here.
No, nothing "wrong" from a technical point of view. From an interpretational point of view, you have to have an argument as to why the dependent variable depends only on aggregates.
This looks like a multivariate folded normal to me...
You might be interested in taking a look at the `coplot()` function and, more generally, John Fox's [effects](http://cran.r-project.org/web/packages/effects/index.html) package.