tttjjj/elt_bench huggingface dataset:
0 is the true value, as no. of articles is a proxy for issue salience. A negative value would have no substantive interpretation (it cannot decrease issue awareness, only not provide any).Concerning the 2nd question, I assume linearity mainly because previous literature models it this way, although I came across some who recommend logging it. I don't see this as a viable method, as I end up with a lot of missing values. Moreover, by looking at the above threads, adding a constant isn't an option either. Using the sqrt seems to approach a linear relationship, but still leaves the 0s.
Because you want a typology, this sounds like a cluster analysis problem rather than dimension reduction.  With your limited data, you could use that and a few basic plots to tell the story - but you're almost into qualitative research methods rather than quant here.
Assuming you have found the MLE of $\\mu$ and $\\theta$ you just need to use this [theorem](http://www.stats.ox.ac.uk/~dlunn/b8_02/b8pdf_6.pdf).
how about the tolerance -e ? what is it effect?
@JMS Thank you.  I think my comment is still pertinent: $BB'$ can be indefinite, but should still not have any negative eigenvalues.  Problems will arise when the smallest of the $\\sigma_i^2$ is comparable to numerical error in the inversion algorithm, though.  If this is the case, one solution is to apply SVD to $BB'$ and zero out the really small (or negative) eigenvalues, then recompute $BB'$ and add $D$.
But yes, it seems like you'll need A LOT of data.
@mathman This is a way to randomly change the data while retaining its absolute spectrum.  Think of the data as a superposition of oscillations of different frequencies.  Randomly shift the phase of each oscillation and reassemble them: you get new data that have many properties of the old.
Ok, this makes sense. Thanks for a great answer.
@Glen-b re: count data, I might have misunderstood your original point. We aren't interested in counts of "y/n" we are specifically looking at did the intervention result in fewer #'s of falls (these "falls" are typically not experienced by 70% of our population. These people were specifically selected for their high number of falls over a long period of time, which appear to be refractory to any other treatment question). I'm conversing with my colleague, to see if we can get the time periods over which these "falls" occured, and thus look at comparing incidence rates pre- and post-.
@chl: I'm using caret 4.98 and R 2.13.2.  I guess I have to wait until the windows version of caret on CRAN is updated.
You would also need to know what a passing grade is.
Given $W$ takes the values ${-1,+1}$ with equal probability, and $Y$ is standard exponential, you just need to show what the distribution of $WY$ is. Which is the same as $X$ being equally likely to take the values $Y$ or $-Y$. Would reading about [mixtures](http://en.wikipedia.org/wiki/Mixture_distribution) help (especially, say the first two equations [here](http://en.wikipedia.org/wiki/Mixture_distribution))? Though you should probably be able to manage from just basic principles.
@Glen_b: If the sample size is large enough, a simple z-confidence interval is still an option: The $(x,y)$-pairs are sampled independently from the same bivariate distribution. Thus, true mean and variance of the considered ratio are constant, although possibly related.
I really think you'd be better off testing correlation, and making a scatterplot. Under the null there is no correlation, so it's a valid test. However, you can use the 10x10 matrix as the input to a Pearson Chi-squared test (`chisq.test()` in R) of independence; the null hypothesis being tested is that the joint distribution of the cell counts in your 2-dimensional contingency table is the product of the row and column marginals.
+1 for the amazing similarity here!
According to http://www.uky.edu/ComputingCenter/SSTARS/www/documentation/MultipleComparisons_3.htm#b23, SAS provide the SIMULATE option for general mixed and repeated measures models. Is there anything equivalent in R?
Sorry to ask, but do you want to test trails and patients are independent? If no, can you please specify what does the Chi-sq test intend to do?
$E_y$ is the expected value obtained when the chain is started from state $y$.
Ok, two questions left;
1) Can there be strong overfit with both algorithms, multilayer perceptron and random forest?
2) How can that overfit when I test my data with 1000 split cross validation?

Would it help much if I post my model .arff file?
As `mpg` increases, you move left on the x-beta axis, making it more likely to fall into category one. And the odds interpretation makes sense: if the odds were 2:1 (i.e., $\\frac23$ vs $\\frac13$ for two outcomes), then 100% increase is 4:1 (i.e., $\\frac45$ vs. $\\frac15$ for two outcomes)
You ask about "prediction model[s]", but I'm wonder if you are really wondering about *explanatory models*. Whether or not a prediction model is useful is based on how easy it is to get the required data to make the prediction, & whether the prediction bears out within a tolerance that is sufficiently narrow for your purposes. Who cares if you can interpret it? Many machine learning prediction models are out & out black boxes, for instance. Interpretability is important when you want an *explanation* for a given phenomenon.
latent.growth.data$monthly_end_with for your problem.
Check ?hlme, it's not the only thing that you're doing wrong.
I've looked up the paper and it seems a bit involved; I was hoping for a function already implemented in a popular software package somewhere, especially because the curve fit I'm hoping for is only going to be for illustrative purposes.
What are you estimating when you weight individuals' scores with weights derived at a group level? Are you intending to estimate eg average white and average asian scores for each variable?  Because this may not be an effective way of doing it.  Ecological fallacy etc.  Do you have racial information on individuals?
Closely related: [Euclidean distance is usually not good for sparse data?](http://stats.stackexchange.com/q/29627/2970) as pointed out by [facuq](http://stats.stackexchange.com/users/25170/facuq).
+1 top quality answers, this stackexchange site is awesome :D
Precisely how is the random variable for the average run length in that formula defined?
You might want to give the names of the books; save people a click. (Also perhaps to mention any affiliation you have with them.)
Any model that uses `lm()` will be a linear model.
@Glen_b Thank you for pointing out the differences: I therefore voted to re-open this question.
Some organizations do have rules.  See the [APA guidelines](http://my.ilstu.edu/~jhkahn/apastats.html) for instance.
Chiming into this excellent answer to add that another meaning of "nonlinear" or "non-linear" is full or partial integration where $y_{t} = y_{<t} + \\text{Other Deterministic Stuff} + \\text{Random Process Stuff}$. That is models of variables that are functions of their past values are also (sometimes, perhaps often) labeled "non-linear".
See http://stats.stackexchange.com/questions/2391/what-is-the-relationship-between-a-chi-square-test-and-test-of-equal-proportions?rq=1
[Kernel functions in common use](http://en.wikipedia.org/wiki/Kernel_%28statistics%29#Kernel_functions_in_common_use)
I'd advise against the wording "Quick question". It implies that you are assuming that because the question can be posed quickly, it will be easy to answer. In this case, and many others, that's not so.
Also, if memory serves, when SPSS does post-hocs it reports the p-values /after/ the adjustment has already been performed.
@whuber I was only given the distribution table with the values 0,1,...,4; I assume that since those were the only values shown, and by the fact I was given a table dataset and not a formula, that the data is discrete and cannot be represented by non-integer numbers. Does that help?
I've answered quite a similar question on [Stack Overflow](http://stackoverflow.com/a/9667122/420055). The on-line help for `rpart.control` provides additional information on how to tune the complexity parameter (`cp`) and its relation to improvement of fit.
Combining (1) and (2) like you suggest can work. I guess your original indicator test if stocks have a fixed change after a shock (which can vary across stocks). My (2) tests if stocks have a decayed change after a shock (can also vary across stocks). My (1) tests if price changes depend on the time interval after the shock. $\\beta_{shortterm}$ would be the coef you get for the indicator that is true for $t\\in\\{151,\\ldots,160\\}$.
I don't know much about support vector machines but this looks mor elike plain old linear discriminant analysis.  In LDA you find the hyperplane in the feature space that best separates the two classes and then use that hyperplane to predict new cases.  So your guess as to what to do with the hyperplane is exactly what is done in LDA.  I am assuming that the term "maximum margin hyperplane" is synonymous with "best separating hyperplane."
I see that preservation of type I errors and confidence intervals would be an issue if the collapsed categories were to be used for further analysis-however I´m more uncertain if the only intent is to show that a grading system is unnecessary complex.
Hi Macro, sorry for not being clear, gbm.object is a fitted gbm. yes, one of the attribute is fit, and I want to know if it is OOB.
@DrWho Point taken.  But you're definitely not talking about "design," then, are you?  Here's another basic point that seems well worth making here: you *cannot* "prove statistically," after the fact with a sample of convenience, that one treatment was better than the other.  The treatment is confounded with the state and all the ways in which the states can differ.  **All** you can do is conclude the data are "suggestive" of a difference.
I mean (a) a bound unlikely to be violated in practice.
Thanks Penguin_Knight, it works very well. Now, is there any way that I show the relationships of two variables regarding the lag with the best "R" in the form of a scatter-plot?
I personally can't discover here any sound question yet. `Any thoughts on these measures? Any other good measures?` seem too broad or rhetorical
@Masood You're right, but when $n>30$ the Student t distribution is very close to the gaussian distribution. Even with n>20, we generally have a good approximation.
Perhaps working _iteratively_ rather than _recursively_ might be better? You have $$e(N,M)=\\frac{1}{2}e(N,M+1)+\\frac{1}{2}e(N,0)+1$$ which gives $$e(N,M+1)=2e(N,M)-2^{N+1}$$ and so $$\\begin{align*}e(N,1)&=2e(N,0)-2^{N+1}=2(2^{N+1}-2)-2^{N+1}=2^{N+1}-4\\\\e(N,2)&=2e(N,1)-2^{N+1}=2(2^{N+1}-4)-2^{N+1}=2^{N+1}-8\\end{align*}$$ and so on giving $$e(N,M)=2^{N+1}-2^{M+1}.$$ Or the difference equation could be solved "theoretically" instead of by iteration.
Wrt. to the Kaggle example: 1. The "internal benchmark" (which seems to be the basis for the "340% outperforms") doesn't say it is the best model Allstate has. Other competitions use reasonably simple and basic models for benchmarking, that may be the case here as well. 2. No domain knowledge: do not forget the amount of both domain and data analysis knowledge that comes in during preparation of the data set. And: I do not know the profession / application expertise of the winner.
@BabakP So does a lot of mathematics play a big role in statistics: but that does not make purely mathematical questions on-topic here. To be on-topic, questions must have an obvious, conventional, or explicit connection to statistics. Otherwise, they would be better off migrated to the math site.
This seems a little beside the point because it's really reflecting the action of a symmetry group on the parameter space leading to an *identifiability* problem. That is, the space of distributions itself may have exactly one global extremum but the *set of parametric representations* of that extremum has $k!$ extrema merely because it has a $k!$-fold redundancy. Another simple example of this phenomenon would be estimating the center of a circular Normal distribution: the angle would be uncertain by an integral multiple of $2\\pi$, giving *infinitely* many extrema.
@King: The reference looks interesting and potentially relevant. Providing a bit of additional context and commentary to connect it to the question at hand would improve the quality of this answer. Welcome to CrossValidated!
@Thomas Migration is easy, but once done, it's very hard to migrate back.  It looks like this question could find a home at either site.  If you're sure you want to move this question to Quant.SE, then just flag it and a moderator will take care of the move.
This is a broad question, can you be more specific towards what you are looking for?
Just insert your data in Microsoft Excel and calculate the means and st. deviations using the built-in functions. Here is the formula if you are interested. It will wive you the interval, in which 67% of population fits given your measurement is normally distributed. http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation.
If they do vary and you are interested  in effects, try anova in R. There are many comprehensive tutorials on the web^ for example https://personality-project.org/r/r.anova.html
@NickCox True, agreed. That is one of the reasons , I was interested in the answer to this everything type of question.
Thanks for the link. If the number of features is too large, the data will be linearly separable and the classification problem is easy - most algorithms will find this separator without "learning anything deep" about the data. But if you dont get a 100% accuracy with new data, then source of the data is not "really" linearly separable - and your algorithms end up using too much information of the training data - overfitting.
I fully agree with this. I am currently training SVMs and ANNs on brain-computer interface data and there are some data sets where SVMs are better and some data sets where ANNs are better. The interesting thing is: when I average the performance over all the data sets I am using, SVMs and ANNs reach exactly the same performance. Of course, this is not a proof. It's just an anecdote. :)
possible duplicate question (c.f. http://stats.stackexchange.com/questions/11609/clarification-on-interpreting-confidence-intervals/11727#11727 ).  It isn't clear how this question is distinct.
Doesn't it just follow from Slutsky?
No problemo. Good luck in your neural network endeavors.
How did you normalize the vectors? Show the new values.
Thanks for asking the question in a new thread, rather than trying to use the "Your Answer" field to do so. You may ask as many questions as you need, so long as they are differentiated from each other, & you spell out what it is that you still don't understand based on the previous answers. Since you have done that here, you are using the site appropriately.
What kind of data you play with? What's the intrinsic dimension of your data? How do you choose how many PC's to retain? Please, be more precise in your question.
Here what would the "posterior class-probabilities" mean? I label the data L and U. Then I for example run a random forest (as you adviced) for classification and choose the output to be "probability". If I have 1000 labeled cases and 100 unlabeled I should get a probability (int his case a proportion) of 100/1100 for the U cases? For each case? or the sum of the probs should be N/(N+M)
Because this is a practical problem, it seems worthwhile to point out that the distributions of the errors in the positions of $L$ and $F$ will be *strongly* positively correlated provided $L$ and $F$ are close, because many of the errors affecting the GPS positions will be common to the two readings. The correlation will decrease as the distance between $L$ and $F$ increases. The answer will therefore depend on that distance as well as on the speed of $L$, its acceleration, and the frequency with which $F$ and $L$ obtain GPS readings. And don't forget the strong temporal correlation...
@Eupraxis 1981 I am not very clear about the term consistency. does it indicate the outcome of say mean here is similar (or equivalent) following a resampling of data. I presume that you are trying to respond in terms of central limit theorem and law of statistical-regularity (convergence).
+1, welcome to the site, @nic, that's not a bad idea. Since you're new here, you may want to read our [FAQ](http://stats.stackexchange.com/faq).
Significance testing for categorical main effects is not any less invariant.  A group may be significantly different from the reference group under treatment coding but not from the "grand mean" effect under contrast coding.
Added two responses to your edits.
@mickey If you like my answer, then you could accept it ...
+1 Fantastic answer!  The first and last paragraphs alone are great advice succinctly put, and the details in the middle show exactly what great graphics can and should look like.
However, it is mostly possible to speed up python code using multiple processes. The multiprocessing and futures module make it easy to do that.
You should look at the `R` `forecast` package, specifically method [nnetar](http://www.inside-r.org/packages/cran/forecast/docs/nnetar) by Rob Hyndman for other insight into how people have dealt with this.
We presented a solution based on Chawla's approach which includes combining SMOTE with random under-sampling. Also, we scaled up the true defaulters 10x, simply by performing random over sampling of the minority class.

Boosting might be worth looking in to as well.
What to say ? Thank you very much for your clarity :) !
There is no closed-form expression for quantiles of the F-distribution. I don't know of any method not involving numeric integration.
You give good answers to lots of questions! That's certainly a qualification for the job, as far as I'm concerned.
MLS , thanks for the answer. FYI, I already know how to complete the square for the integral.
What kind of "anomaly" would you be looking for?
+1, for a question that I think is important, and clearly & fully stated. One thing I will say right off is that, re #3, if you are finding differences in the pattern of results across the two different levels of IV1, you probably have a type II error in finding the interaction 'non-significant'. Either the pattern is the same at both levels of IV1, or the interaction obtains.
I believe the correct syntax for the arima simulation is `iman.sim <- arima.sim(model=list(order=c(0,1,1), ma=-0.9), n=100)`.  I think the `d=1` is ignored, and you're actually generating an MA(1) process; the first order autocorrelation of roughly (+/-) 0.5 would support this hypothesis.  I agree, however, that `arima.sim` is the way to go.
Hard to see how a person can pick up logistic regression when the Chi-Square Test is posing problems.
In fact I just made an example where the biggest pairwise correlation was 0.38 yet there was perfect multicollinearity (among 11 predictors of length 100); much smaller values would be simple to achieve.
Thank you Frank - this is very helpful :)
Why do you want to do this? I am not sure it is a good idea, but perhaps it is.
Have you tried to write the equation for $y_t$ by substituting for the analogous expression for $y_{t-1}$ and then go on, recursively backwards? Then insert what you will have find in the formula of the OLS estimator, and all will become clear then.
@Andy, sort of, I was suggesting using a one class classifier for novelty detection, but a standard multi-class classifier for performing the classification of observations that are not novelties.  I think that may work out better as one-class classifiers are good for novelty detection, but are not likely to be as good as a discriminative classifier for performing the actual classification.  Libsvm is an excellent bit of kit, so you are off to a good start.
@cardinal I look forward to what you have to say.  If you don't have time for a thorough response, even a sketch of what you think might be the "best idea for these data" would be most welcome.  I can guess where you're going with this: the data have been ranked, a process which creates dependencies and should require me to defend a likelihood derived without recognizing the potential effects of the ranking.  It would be nice to see an estimation procedure with sounder justification.  I am hopeful, though, that my analysis can be rescued by the sheer size of the dataset.
The practical approach you seem to advocate here boils down to - when both covariances and correlations are warranted - "try both and see what works best". That pure empirical stance masks the fact that any choice goes with its own assumptions or paradigm about the reality which the researcher ought to be aware of in advance, even if he understands that he prefers one of them fully arbitrarily. Selecting "what works best" is the capitalizing on the feeling of pleasure, the narcomania.
Probably, but it's hard to see why. For a generic finite-state Markov Chain of up to, say, 1 million states, this can be done in about three to five lines of code in either language and will run on a typical laptop. Maybe you can update your question and tell us about the problem you're actually trying to solve. This is likely to yield a more useful answer. Cheers. :-)
@whuber: Thanks whuber!. I've looked around the site, but to me I am still finding it a bit challenging to understand what is really meant in the context of my own research.
Please show more about your model `fitted_data`
It's a feature indicating the presence, absence or "weight" of a word in a document (i.e. a "bag of words" model).
You can't do anything without assumptions, Nick!  But regularity (such as smoothness of the function) is not necessary: it can be deduced from the data, *at least on the scale at which the function is sampled.*  (Stationarity is a much milder assumption than smoothness.)  You are correct that largish samples are needed, but much can be learned in 2D even with 30-50 well chosen sample locations.  The literature is large; for instance, most of the issues of *Mathematical Geology* are devoted to this.  For a rigorous introduction, see Cressie's *Spatial Statistics.*
I assume what is being looked for is double exponential smoothing. I have no idea how to do it in open office or even R without writing out forumala
How having AR process as a predictor relates to the variable that I want to predict? That's the bit I'm missing.
The evaluation of the middle integral is incorrect: it's zero, not a logarithm. The problem actually lies with evaluating the two limits implicit in the infinite integrals.
They're both testing independence against dependence, but in different ways. DW explicitly tests lag 1 autocorrelation, the runs test is somewhat sensitive to that, but also responds to other forms of dependence. It's a bit like comparing a Jarque Bera test against a Shapiro-Francia test when you're interested in normality; they both are tests of normality against non-normality, but they respond differently to different aspects of non-normality. If you're *specifically* interested in what a more specific test looks for, that's a good thing, otherwise it might not be.
I am not sure what the data means to you.  If the variables are independent, then you may need to look at exact agreement for each of the variables.  You may find that some of the groups (e.g., variables 1-5 have different agreement than variables 6-10). If some frequencies are rare, then simply percent exact agreement might not be meaningful and you might want to try Cohen's kappa. If the variables can be grouped and if can be summed within groups, then a simple correlation might be meaningful to you. I guess I don't understand the project yet.
That's a really interesting observation @whuber. The argument breaks down for the Bernoulli(1/2) because then $(x_i - E[x])^2 - \\sigma^2= 0$ is constant. Strangely enough, it works for Bernoulli(p) with $p \\neq 0.5$. The Bernoulli(1/2) counter-example is a great illustration of why it's important to check those annoying regularity conditions instead of just saying "the central limit theorem tells us that under some conditions ..." like I did.