tttjjj/elt_bench huggingface dataset:
It is hard to say what the problem is without seeing what you did. ARMA(2,2) is just given by  X(t) =r$_1$ X(t-1) +r$_2$ X(t-2) + e(t) + a$_1$ e(t-1) +a$_2$ e(t-2).  Choose initial values for X(0) and X(1) generate e(t) for t=0,1,2,3,... using a normal random number generator with mean 0 and a fixed variance s$^2$. Then just use the formula to recursively generate the X(t)s.
Honestly, I don't see much of a problem with using the sum of 4 Likert values as if it were a continuous response value, and using ordinary analyses. I suppose if there is a very strong tendency to answer at one end of the scales or another, that could create issues. But otherwise, it's just a score. PS - the "fractional factorial" tag definitely does *not* apply to this question. With a product of primes in the full design, there's no way to fractionate it.
Can you add some more details to your question? How many timepoints do you have in your panel? Often you can tell your software that your data have this structure & it can take care of it for you w/o your having to explicitly add terms to the model.
@ahfoss: (+1) to your latest comment. *Yes!* You got it. As for examples: (i) it shows up in censored data problems, (ii) classical applications like hidden Markov models, (iii) simple threshold models like probit models (e.g., imagine observing the latent $Z_i$ instead of Bernoulli $X_i = \\mathbf{1}_{(Z_i > \\mu)}$), (iv) estimating variance components in one-way random effects models (and much more complex mixed models), and (v) finding the posterior mode in a Bayesian hierarchical model. The simplest is probably (i) followed by (iii).
Linking 1-D CDFs makes me think of [copula](http://en.wikipedia.org/wiki/Copula_%28statistics%29)s. Not sure if they'd be any use to you.
Ah, you say of a situation when a later link on a dendrogram lies lower than some earlier links. This "inversion" naturally occurs not very rarely with regular centroid/median methods, you probably know. It's no wonder. In such a case they recommend to reset the peculiar $\\delta_q$ to the highest link among local subclusters, see [example](http://stats.stackexchange.com/questions/26769/cluster-analysis-in-r-produces-reversals/26775#26775). So, this can be overcome.
Regarding JMP - If I'm not mistaken, there are trial versions and academic licenses that are within the reasonable.
Dear Nick, thank you for your answer. I have 54 monthly data and use with the 11 dummies 4 additional parameters. Is that ok or unstable? I get at least reasonable results.   What do you think about grouping the dummies into quarterly dummies to aggregate the monthly effect, if I have similar coefficients for several months in a row
I think the question is too vague to answer. For instance, how may we interpret "along with all the details of how the study was done."? Do the details include standard errors, or an assessment of model-fit in general?
+1 for using a binomial mixed effect models. Alas it was rejected as "not a standard method". If you can give me some links to papers where this approach is used in a meta-analysis setup, you would help me greatly. Thank you in advance.
Remember, the Welch formula is an *approximation.*  [Simulation studies](http://biol09.biol.umontreal.ca/BIO2041e/Correction_Welch.pdf) indicate the "Welch correction becomes too conservative when sample sizes are strongly unequal," which is the case with the A-D comparison.
Maybe you can clarify the wording of the sentence "Provided $B$..." since I don't believe the first condition implies the second if $A>0$ and $B < 0$, for example. (I'm likely just misunderstanding what you meant.)
If you can write it as a function, you can probably write it in R. If (as seems likely) the pdf involves integration, have a look at [this](http://stat.ethz.ch/R-manual/R-devel/library/stats/html/integrate.html), or try `?integrate` from R.
This doesn't get much at the substance of your question, but I think the `transform` function is probably a lot more efficient for adding variables; i.e., `DF2 <- transform(DF1, s = sapply(etc))`
Good question. Even more importantly, the data will be strongly positively correlated over time: that will have a more profound effect on the answer than the variation in the measurement uncertainties.
[Differentiation](http://en.wikipedia.org/wiki/Differentiation_%28mathematics%29) vs [Differences](http://en.wikipedia.org/wiki/Finite_difference)
Ah! You hit it dead on the nose. I did indeed do this via software, and the [MATLAB help page](http://www.mathworks.com/help/matlab/ref/mldivide.html) about the `mldivide` function/operator does indeed explain the behavior. Thanks for the explanation, and your patience with my ignorance :-)
Perhaps it is best to qualify the method further: Hahn and Shapiro (as referenced above) advise that caution should be exercised, especially when the sample size is less than 200 - and recommend that this be followed by further verification, such as a frequency table that compares the fitted distribution with the actual data.  But in my view it is a useful method that *can suggest* where the data might lay within a spectrum of possibilities.  I have used it on data sets not smaller than roughly 3000 and have built it into computer simulation software where it has proved useful.
You should really explain why
I don't think this is a sensible question. Stata's `margins` deal with everything you need. There's no effect of `age^2` that would be separate from the effect of `age`; there is simply a non-linear effect of `age` that `margins` treats perfectly fine for you.
It's not at all clear what you're actually asking for here.
@MattMunson There are many different types of Neural Networks, which internally use different algorithms to reduce some form of error metric. Can you say any more about the structure and algorithms that the NN in question is implementing. This might be very relevant in giving insight into how it behaves and the effect that its parameters may have on performance.
@PierreLison:  Good idea.  The update is just collecting the counts of $o$, but the function that turns those counts into a maximum likelihood value of $F$ might be a little bit tricky.  Feel free to ask another question if you end up going this route…
I have the same issue... have you solved it?
I don't think that the problem is convergence. I think the problem is that the sampler is poorly performing because of the high degree of correlation at the multiple levels of the model: $\\hat{R}$ is low, while $n_{eff}$ is low relative to the total number of iterations. I would suggest just computing the posterior directly, for example, over a grid $\\theta, N$.
**Another** important idea that you should take away from your course is what is sometimes called the [Law of the Unconscious Statistician](http://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician) which says, in this instance, that $$E[XY]=\\sum_i \\sum_j x_i y_j p(x_i, y_j).$$
This might be along the lines of what you intend http://en.wikipedia.org/wiki/Computerized_adaptive_testing
Alright, I have measured dendrite length (apical and basal) which provided me with 3 values, apical dendrite length, basal dendrite length and total dendrite length. I've also measured 5 different electrophysiological paramaters within the same cells. My RQ is: Are the electrophysiological parameters related (correlated) with the dendritic length. This yields 3x5 comparisons (3 length parameters with 5 electrophysiological). I have found some significant correlations, as expected. Now i want to find out whether the significance of these correlations must be corrected.
What does the production environment support?
@Macro, excuse me for my lack of knowledge of the terminology here, but what I would like to do is to use a random variable to represent a frequency distribution, and I use an instantiation of  this distribution to represent a state of world (the world can take one of many possible states, and each state corresponds to an instantiated frequency distribution). Of course, this random variable has a distribution, which would be like a prior distribution over all the possible states the world can be in. I hope that clarify my question a bit.
Thank you so much for this information, 
recently , i've tried to use , DBSCAN clustering algorithmg (with some modification in code)  for detection anomalies in Times series signal, FYI : the anomalies in this signal is a jum from a normal stat to abnormal ) 
anyway! i WILL read this articals now, hoping, find what i'm looking for,
Thanks for your thorough look! I agree, it looks like a but. I'll keep those tricks in mind next time.
By A*gender I meant to suggest you include an interaction of A and gender.  This term would allow you to see how the relationship between A and B differs among men and women.  You should go beyond the simple question of: "Is the relationship between A and B statistically significant?" and try to learn everything your data can tell you about this relationship.  The model you use to predict B depends on, among other things, on what B is... continuous and normally distributed, dichotomous, ordinal, etc.  Please give us statistical output and then attempt to interpret it.
I agree with @jthetzel, I'm not sure that I quite understand your question. An unbiased estimate is one where there is no bias and where you can rely on your sample size, lean back and let the statistics do the work (...and yes, it probably never occurs in reality where you always have some type of bias). I try to keep it simple: bias is to me always a systematic error but as I said - there are far to many books on this issue and unfortunately many are written by people who barely grasp statistics. Ask an epidemiologist about effect modification - many (most?) think of it as a kind of magic...
Well, `doRedis` can only do redis stuff; huge input is not the only reason for HPC calculations.
Let I use UCT selection strategy. Currently I'm in a stored node **v**. So my next node will be **k = argmax_i(v_i + C * sqrt(ln(n_v)/n_i)** But what if n_i = 0, I mean i is not stored.
I took the liberty of defining your $z$ using the standard signum function. That frees up the symbol $y$ for its usual role.
I'm not sure I understood, this is for p(A or F),like p(A) or p(F), and what about M is my answer correct? :)
Based on the abstract the Zou paper does look dead-on as one that would address your problems.  If the fact that the 3 scores share some components makes you leery of using Fisher's Z transformation, the same problem would seem to apply to a regression approach.
Can you edit this to provide more information? For example, what software are you using? As it stands, the question is too vague to be answerable. You may want to read our [FAQ](http://stats.stackexchange.com/faq) or [this blog post](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) regarding how to ask a statistical question. Note that, if this question cannot be made answerable, it will need to be closed.
Good point.  I got an estimate of the variance of s^2.  Taking the square root gives an estimate of the standard deviation of s^2.  But you answered the actual question which was to get the standard deviation of s. I would assume that for practical reasons you too woulde replace σ with s to get an estimate using the formula.
I just experimented and found out that using SGD gives quite good results for my specific problem. I know that it's rather for convex type of problems, but I'm more or less satisfied with the results, except this sudden divergence sometimes. Do you know any better online learning algorithms for this kind of problem?
When the best fit has $b\\gt 0$, if you want to constrain $b\\le 0$ you can set its value manually.  For instance, if you set $b=0$ you will get a quadratic fit.  For smaller values of $b$ you *must* set them manually (according to some arbitrary criterion).  Incidentally, to find decent starting values simply regress $\\log(y)-2\\log(x)$ against $x$: the intercept estimates $a$ and the slope estimates $b$.
@whuber I now understand why the two numbers were different - they answer different things. (I wrote a quick program to just try many rolls and see, and both of your numbers match exactly.) But can you explain how you calculated your numbers?
This is very interesting, thank you. But could you explain how it answers the question?
The crux of your reply is to check whether the "model is conventional and parsimonious ... and it makes the data look reasonably typical."  How precisely does one check these things?
The mean you described (the arithmetic mean) is what people typically mean when they say mean and, yes, that is the same as average. The only ambiguity that can occur is when someone is using a different type of mean, such as the *geometric* mean or the *harmonic* mean, but I think it is implicit from your question that you were talking about the arithmetic mean.
@lennon310 Almost all kernels evaluations are dependent on $d$ since they typically involve an inner product. This is true for all common kernels (linear, polynomial, sigmoid, RBF).
[This blog](http://jepoirrier.org/2012/05/16/visualizing-categorical-data-in-mosaic-with-r/) has a pretty nice example. You can also augment two barplots to show the summed counts of Factors A and B as a multi-panel graph. For diagnostics, perhaps you can change the raw count of the mosaic plot to be residual.
Can you describe a little why cross-validation would be a solution? Would cross-validation allows certainty with which some item belongs to class become the probability of an event taking place?
see my edited answer.
I recommend the excellent package 'PBSadmb' available on CRAN.
You lose channel capacity in information transmission between you and any other people when discussing statistics. This CV post has some info: http://stats.stackexchange.com/questions/10810/who-first-developed-the-idea-of-sampling-distributions
@peuhp There seem to be two senses of the word "dimension" in operation here: one is the dimension of a manifold (coordinatized by the parameters $\\theta$, of dimension $d$) and the other is a dimension of a space in which it is embedded (with coordinates $\\eta$, of dimension $s$). The latter was likely meant in the C. Robert quotation--but whether that is the correct interpretation depends on exactly how Robert defines "dimension."
Why do you think bootstrapping requires normal data? Also, what exactly are you trying to show? "Not random" is not a testable hypothesis, without specifying what distribution you want to show they are not random on.
You don't *have* to, but it make make the data better behaved or more naturally interpretable. This thread is worth reading: [In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?](http://stats.stackexchange.com/questions/298/)
(+1) Good question!
So the two factors in your ANOVA are pre/post and treatment, for a 2 x 3 design?
@Matt Yes, I was thinking of this kind of stuff. Put it as an answer!
@PolyGeo Is it really necessary to have a software platform in mind in order to understand and answer this question? I really don't think so: it's perfectly clear.  Matt, although your question should be of great interest to anyone analyzing raster data with GIS, if you don't get much love from this site consider flagging your question for migration to [stats.se] where the community is well qualified to address the statistical issues you raise.
For the problem of 1-dimensional Kernel density estimation, the choice of Kernel is known to not matter as much as the bandwidth. For this problem, the Epanechnikov kernel has been shown to have asymptotic minimax properties but people often use Gaussian kernels anyways because it doesn't really matter much. I would regard this as a different sort of thing than the much broader "kernel methods" used in machine learning for regression/classification where e.g. kernels might be engineered to reflect crucial assumptions about the data.
@chl Median polish can work with as many factors as you might care to handle.  Tukey does some three-way examples (by hand!).
...More) So what I'm proposing is that if the performance measure you've adopted doesn't let you measure the performance, then you need to consider an alternative.
@whuber, yes, I have edited my post to clarify that I'm asking a statistical question about simulating the data for a random effects model.  I'm not sure I understand exactly what you're asking.  What I am trying to ask is how I would simulate data that reflects the results of the mixed model I ran on my preliminary data.  I think it is a statistical question because I need help understanding the way that the fixed effects, random effects, and correlation can be integrated into the final model.  I was originally asking for code because I often find examples easier to understand.
But those are different things. One thing is compute the value of the likelihood for some value of the parameter (this gives you a real number). Another thing is, given a value of the parameter simulated from the prior, to simulate a synthetic $X$ from from the conditional distribution (ABC needs this). The duality likelihood x sampling distribution is important here.
This is excellent. Thank you for the well formed answer.
@JDav That's the question ... what's an appropriate distribution comparison for this type of test. Even when I'm using permutation testing I need to know a reasonable way to compare the distributions.
Regardless of whether the distribution of (X ,Y) is bivariate normal or not the Pearson correlation is a measure of degree of linearity.  The probability distribution for the sample estimate will depend on normality.
@Jeromy thanks, posted http://getthedata.org/questions/399/where-can-i-get-real-data-of-big-network-topology
@Sergey, I only looked at the computation of the number of parameters in my answer. For completeness I should have considered how to compute the "size" of the data set too, as you are interested in BIC. To be honest, I have only seen a derivation of BIC for independent data, but my guess is that the correct "size" of the data is the length of the data series. I am not sure what you mean by the size being 2x100. If you want to comment on that, you have to give a more precise description of how the data are encoded as a 2 by 100 matrix.
@charles my apologies. It is indeed not necessary to scale the features in using random forests. See: http://stackoverflow.com/questions/8961586/do-i-need-to-normalize-or-scale-data-for-randomforest-r-package
http://en.wikipedia.org/wiki/Winsorized_mean
@MichaelLew, I'm not sure that the p-value means these things on its own. *In conjunction w/ N* (& specifically, holding N constant) a smaller p will correspond to a larger discrepancy b/t the null & observed. Even then, that's more of something that can be inferred from p rather than something p *means*. It is also true that w/ larger N observed effect sizes should be closer to true ES's, but it's less clear to me what role p plays there. EG, w/ a false null, the true effect could still be very small, & w/ large N we would expect the observed ES to be close, but p could still be large.
Regular OLS regression only assumes that the DV is continuous, not anything about its shape. It assumes that the error (as measured by the residual) is normal.
(+1) A description of this and other tests can be found in [this blog](http://normaldeviate.wordpress.com/2012/07/14/modern-two-sample-tests/)
Based on the rest of the passage, I think the author was clearly referring to the sample mean
In a soft-margin classifier, you accept some amount of examples times normalized distance within the margin. Duplicating the $X$ in the middle may make the penalty too high, greater than the acceptable threshold, so that you have to decrease the margin by a lot at the old position. This makes the new position a better soft-margin classifier when the $X$s in the middle are duplicated.
How are these values generated? Do they come from a uniform distribution?
@crucifiedsoul - your code looks fine to me.  Next time I'll give you Java code!
I think the point of the OP is exactly that the green and the magenta lines are the correct ones, and he wants to find them. The red and orange lines result from assigned the readings to sensors in a somewhat random manner.
You seem to have answered your question in asking it. As you have framed it, the answer seems to be 'obviously'. But you have to be careful because the conditions you have set up for your question will be tricky to be confident of in practice.
This question appears to be off-topic because it is about how to do something in SPSS. It may not be welcome on StackOverflow either, as it is a "show me the code" question
Above comment about the data sheet example I used does not show in the table-shape, but I hope it is possible to make sense out of it. :)
If we assume that the counts of women and the counts of men in a year are Poisson distributed with different means (a reasonable assumption here) then at the end of the year we have a total number of people that arrived.  *Conditional* on this total, the two counts must therefore, as a matter of mathematics, be Binomially distributed.  That's just what happens when you condition two Poisson variables on their total.
Could you post the reference for the second method (Track an online approximation of a fixed...)? Thanks.
It's probably not but I already indicated the test isn't the best way to tell, just look at the data.  Regardless, you've got a resonable non-parametric solution.  To summarize, there is no good "test" for normality because even with very normal data it will always fail if N is high enough.  Nevertheless, you have a small N and the test is reflecting what you would see in plots anyway.
Can you explain how your reparametrization encourage sparsity? It rather seems to *guarantee* quite the opposite.
Please update the title to something more informative and, if possible, give a citation to the paper in question. The calculation $U(i)$ is an [exponentially weighted moving average](http://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average) of $Z(i)$.
See http://stats.stackexchange.com/q/67533/9964, which is basically the same question. The short answer is I don't think there's a closed form but there are approximations or computational ways to compute the PDF.
Is this about PCA or FA?
This question is very much like your previous one. Please merge them into one question.
I am talking about variables
Haven't done much survival, so not going to answer, but I believe de Vries' code example is incorrect for time-dependent data. `Surv(time, event)` would be for a more standard analysis where you just have one summary line per person.
A related phenomenon is discussed at http://stats.stackexchange.com/questions/14500/. You seem to be performing an *ad hoc* version of "stepwise regression," which has been [extensively discussed](http://stats.stackexchange.com/search?q=%2Bstepwise+%2Bregression) on this site.  There is practically a consensus that this method is best avoided (or used with care by experts in particular situations); there are better model-building techniques available.
Care to explain your recommendation or acknowledge any relationship to the author?
@Max And no, you sample from steps 1 and 2 and then subtract 1 from 2 to obtain one sample from 3.  You do this M times to obtain M samples.
Any idea of syntax for a non-smoothed categorical random effect?
@JohannesEbert: three points come to mind. First, I would really recommend using the first PLS component rather than the first PCA component in your regression (or maybe use PLS regression directly). Second, in either case you will be *much* better off with an iterative method to find the first PC rather than using the standard eigenvalue method; try iterative methods that do not precompute covariance matrix (maybe R can do it for you, this I don't know). Third, if you do want/need to use subsamples, try selecting them randomly (and use several random subsamples, hoping to find the "same" PC).