tttjjj/elt_bench huggingface dataset:
Very true. Sorry to be misleading. I added the one described in the question to the left. As I mentioned, I want to know how much is a moderate deviation from normality in the different samples I have. Hopefully with two images I can get even better feedback.
@alwin: caracal's answer explains the difference well, but it should be noted that this is more than a technical problem of getting the softare to agree; you should think about your scientific questions and decide which is more appropriate.  In most cases that I've done, the Type I or II (which are the same here, I think) is more appropriate than Type III, but you should think about your questions and decide for yourself.
"With your exemple you can see that in the thrid process you take 2 measures, this is likely to be more reliable then taking only  one measure."

The SEpred is always equal to or higher than the other two. My understanding is that this is because taking the test twice results in an additional source of measurement error.
Thanks @michael!.
I should have been more clear on that, you're right. I changed the answer to explain that fact.
@iliasfl.Exactly it all depends on your definition of Big Data. If by big data you mean Voluminous data,R now has packages for distributed computing like `ff` for using R on virtual machines, `RHadoop and Hadoop streaming` for running R on a hadoop cluster.

If by big data you mean variety not volume, R can handle almost any kind of structured and unstructured data.

The only problem in R is , you can't make an executable file as you can do in python, so whenever you are deploying a tool for external client, you have to show source code and install R on the client machine
Ah ok that makes sense. I wasn't aware there was a `ifelse` statement available in JAGS. Does this seem theoretically sound? I think originally I was confusing myself and assuming my `mu[i]` node was the expected value rather than the scale parameter.
It is also not bad to take the Extended Kalman Filter in this case.  It is integrable in time up to step of measurement.  If the discrete Kalman filter assumptions are valid (linear, Markov, uncorrelated noise,...) then they are valid in the case of the EKF.  Sometimes the EKF formulation is more useful because using appropriate numerical integrators (Runga-Kutta, Adams-Bashforth-Moulton) can be more accurate and easier to set up and use in continuous domain than in discrete (z-transform) domain.
And so if one wanted to they *could* train the network using regular gradient descent rather than back prop and it would optimize in the same exact way, except it would be much less efficient?
this may be problem-specific though ...
The hard part is determining what constitutes a "communication pattern" and how to evaluate how similar two such patterns are.  What do you propose for this?  (Your discussion of "features," "patterns," "data items," and "similar" is too vague to evaluate.)
thanks! Very clear answer.
In my experience *leave one out* cross validation almost always results in too little regularization. $k$-fold cross validation nearly always works better.
How can I account for the autocorrelation and heteroskedasticity at the first level or student level? In other words, I have repeat observations for each student (over several years), so the student level errors are going to be certainly autocorrelated and potentially hetoroskedastic. I should probably modify the above code don't I? Or, does the above command address both of these issues? Please let me know. Thanks.
Thanks @Peter. I appreciate your help. However, I carefully looked at the problem. In usual regression we may look at the predicted values. But in count data it might better to look for counts. Like, true value is 0, but predicted value is 0.16. Also true=1, pred=0.66. Hard to make any conclusion. Anyway, when I used column sum then it gives me the average. But I should have look at covariate specific predicted values. Like, if covariates, say x, y, z, are low how each model works.And compare them with true counts. Now ZIP, Hurdle and ZINB became superior.
+1 to @nico for providing a fully working empirical way to test different options. A very pragmatic way to test approaches, esp with the kind of tools most of us have at our disposal (R). I think the OP wants a confidence interval for his estimate of the median.
Point of clarification required - Is $K_4$ defined to be the point-wise multiplication of entries in $-K_1^2$ and $K_2$ or the matrix product $-K_1^2K_2$?
@Sarunas i'm not sure, but if you ignore the design, the burden of proof is on you. maybe [run it by bls](http://data.bls.gov/cgi-bin/forms/cex?/cex/csxcont.htm)?
Good question, but sorry currently I cannot tell how much would it affect your model if you are still using the batch gradient formula in the answer, or approximating by applying the matrix form directly: eta^(-alpha)*X(Y-X'beta_n) where X, Y are your new batch samples
Sorry about the algebraic error: the right hand side should read $\\frac{a}{a-b} \\left(1-\\frac{b}{(a-b)X+b}\\right)$. Although this is undefined when $a=b,$ the ratio on the left hand side then reduces to $aX$. I suspect you can do better than second-order error propagation (the "delta method") by considering the distribution of $X$, but that depends on exactly what $X$ is measuring and how it is distributed in your population.
@whuber No one said that there was any important relationship but the OP seemed to ask for any connection between the mle and the t test.  I am not sure that the edited title is really what the OP intended.  That is why I made the comment to the OP.  I was trying to show how the t statistic relates to the maximum likelihood estimates for the mean and variance of a normal random sample. The sample mean is the mle for the population mean of course and the mle for variance is V=∑(X$_i$-X$_b$)$^2$/n.
The first thing to do would be to count how many users made comments.  The second would be to read (some of) them to see what kind of information they contain.
Glad to help, @Helene. If this answer provides the info you need, consider upvoting it & accepting it by clicking the check mark to the left of it.
No. Only one of them needs to be. Let $X$ be zero with probability $1$ and let $Y$ be any finite-valued random variable.
I know. I just have habbit of leaving question "open" for at least a day after the last answer is posted.
@mbq Sorry, I am somewhat confused. Do you mean that I should take the log of the prediction (SVM's output) before calculating the correlation coefficient?
I'm looking for a value of p that will generate n successes with probability x=0.05.  Values of p below that are not in the confidence interval because 95% or more of the time you will not see n successes.  Solve x=p^n; log(x)=n.log(p); p=exp(log(x)/n).  1 is obviously in the interval because it is the maximum likelihood estimate after n successes; and is also the upper bound allowable for the parameter p.
Incidentally, JackOfAll, are you aware of how the process of upvoting and accepting answers works on this site?  It's a valuable aspect of creating great questions and answers, so please spend a few moments reading our [FAQ](http://stats.stackexchange.com/faq) and act accordingly on the questions you have already asked.  In particular, any thoughtful, well-crafted useful answer, even if it's not the best in the group or only partially addresses your question, deserves your vote in recognition of the effort on the part of the respondent.
I think I understate the question. My point was to: fit Gaussian (2 parameters) and calculate $X^2$, then fit $p_{final}$ (2 parameters for Gaussian + (t+1) parameters for the thresholds) and calculate its $X^2$. Calculating smoothness (e.g. in the form of $\\sum_{s=0}^{99}|p(s+1)-p(s)|^2$) may be interesting but then it is important to check underlying assumptions and etc (e.g. for tests with a lot of question of for 2 points there may be fairly high "initial" jaggedness). If one has access to raw data (i.e. all answers, not only total scores) then there is even more room for testing...
I don't see documentation for svm.weights, do you mean svm.coef_ ?
There could be fewer modes than time segments.
> *in that values of B above a threshold will change C* Do you know that threeshold in advance or does it need to be estimated as well ? Also, when you write B*t do you actually mean $B_t$ ?
It isn't clear to me what is going on here. What do "networks" have to do w/ anything? Can you say more about your situation, your data, & your goals?
I'd love to see this loss minimizing in clustering, kNN or random ferns...
Whether the model is correct or not, you can still generate confidence intervals for the predictions it gives. The confidence bands will just further evidence that the model you have isn't a good one. The question was specifically how to construct confidence bands for the given model.
I wholeheartedly disagree. We should cover statistics software here too. How many R questions are there?
Now addressed in mice 2.16 (will stop with an error message).
You seem to be saying "what if the data aren't paired?". If values are positive, you could consider shifts in the mean of the logs, perhaps (since on the log scale the mean difference of pairs and the difference of unpaired means should both estimate the same location difference).
@steffen "So you basically say, for every iteration one adds the random noise of interest and calculates the resulting SSE by rerunning the complete cluster algorithm including the noise datapoints ? " is the correct interpretation of my answer.
@Andry: see my edit.
And could you please say what standard variable selection techniques are?
@cardinal, I am not forcing a nesting, merely using a dimension consideration. If we have a $q$-dimensional subspace you can always choose $w_q$ in that space such that it is orthogonal to a $(q-1)$-dimensional subspace. Then you fill up the $w$-basis in any way you like.
General answers have also been posted in the duplicate thread at http://stats.stackexchange.com/questions/91750.
Your point about collinearity is very interesting, and seems to be a possible explanation for the inconsistent results between the "whole sample" and "no meds" sample (not to mention the adjusted and unadjusted versions of the whole sample analysis) results when ADHD was the DV and bodyweight was the IV. Interestingly, however, this wasn't an issue when bodyweight was the DV and ADHD was the IV - those results were remarkably consistent. I will have to try to figure out why that is, while I also try to wrap my head around the Simpson's Paradox. Thank you for your insights and suggestions!
That's not the reason that the ratio of independent normals has no (central) moments. Indeed $\\mathbb P(Y = 0) = 0$ in both cases.
I think survival analysis is the way to go here: it deals with time to events (in your case: reply). Specifically the *Kaplan-Meier* estimate is what you want. This gives you a new view of your data set, and inference is more natural in this setting.
You said "**One** of the lab's data is non-normal, therefore I am looking for a non-parametric test." -- however, the usual assumptions nonparametric tests tend to make is that the distributions are the ***same*** (apart from location shift). If the shapes are all over the shop, nonparametric tests might be affected as much as ANOVA. Even the common bootstrapping-of-residuals (or a number of other resampling approaches) will be screwed up, because they rely on the ability to swap observations across groups. Resampling within groups might be used (one that resamples the rows of $[X|y]$ say).
Proof alternate #3 has a beautiful intuitive explanation that even a lay person can understand. The basic idea is that the sample mean is not the same as the population mean. Your observations are naturally going to be closer to the sample mean than the population mean, and this ends up underestimating those $(x_i - \\mu)^2$ terms with $(x_i - \\bar{x})^2$ terms. This is probably obvious to most people but I never thought about the "intuition" as to why the biased sample variance is biased until now. I only learned the formal proofs.
I think this answer is __wrong__. The response variable is a count variable and dichotomizing it will loose information. You coud try with a usual linear model, maybe with a transformed response variable (such as log).  But you should also make plots of the data, and diagnostic model plots. If you dont understand what this means, you need local statistician help.
Thanks for the links, I will have a look at them later.
You have probably read through this, but just in case you haven't, this is the one of the top review articles on HMMs http://people.sabanciuniv.edu/berrin/cs512/reading/rabiner-tutorial-on-hmm.pdf
This (famous) class of strategies is a very refined way to go broke fast. Sending me $100 a month is less refined, but will send you broke so much more slowly that it will seem profitable by comparison.
If you are trying to fill in missing target values (to generate data) then instead of filling in the most likely, you could fill in a random value whose probability distribution is given by your model. If your model says that target value A happens $35\\%$ of the time, then you fill in A $35\\%$ of the time. In some situations that is much better than filling in the most likely value $100\\%$ of the time. E might be the most common letter in English but not many sentences consist of only Es.
Rearranged the math a bit for you, making it a little clearer; hope you don't mind.
Although the description of K-means in the first paragraph is not wrong, it may mislead some people into equating this method with Voronoi partitioning *based on the original data.*  This is not so: the partition is based on the locations of the cluster means, which might not (and usually will not) coincide with any of the original data.
Isn't it the expected value of an outer product of the score with itself?
I don't know the exact answer, but I do know that estimating very small p is tricky. This is yet another reason to emphasize effect size rather than p value.
Thanks a lot for your advices. I am trying to follow them but get a problem. 
I am currently using ARFIT matlab toolbox (but I think my problem would be the same with an other program).
 It seems that it is not possible to get a global fitting of my 16 subjects and 8 time points by subject. Indeed, the fact that i get more subjects than number of time points is problematic. Is it true?

Do you have some other advices? thanks a lot for your help
Lovely. Perhaps this is the reason this data is not decipherable :D
Can you summarize the solution to the OP's problem provided by the paper? Links sometimes expire and then this answer will become less useful.
@HomunculusReticulli if you fit the model with `impedance` as a continuous variable it will return for you a monotonic function of the probability of state change as a function of `impedance`. You can then just solve the equation for x%, since it is monotonic that will be the cutoff point.
I would suggest you [this article](http://onlinelibrary.wiley.com/doi/10.1002/widm.12/full). They do something very close to what you described.
@PeterFlom Very good point. Using both pre and post scores in a paired model would be superior. However, in a pinch, one could naively find a relationship between change scores and QoL and infer that QoL was associated with test score differences.
Of possible interest: [Effect of switching response and explanatory variable in simple linear regression](http://stats.stackexchange.com/q/20553/930).
Great! The first important part to me is that your retention criteria use $\\bar{\\lambda}^{r}_{q}$, i.e. a specific cut-off value for each factor $q$. The questionable sentence *“parallel analysis suggests that only factors with eigenvalue of 2.21 or more should be retained”* equals $\\forall~q~\\bar{\\lambda}^{r}_{q}=2.21$. This is impossible. For principal component annalysis eigenvectors add up to $p$, for factor analysis to $< p$. One single $\\bar{\\lambda}^{r}$ irrespective of $q$ exists only for fully uncorrelated data ($n \\rightarrow \\infty$) and then it is either 0 (fa) or 1 (pca). Correct?
@bnaul: No idea how people work in your field, but contemplating differences on a 10^-50 or 10^-100 level?? Even with n=1000 I image that would be hard to back up with actual data. Hope you people can sleep well ;)
I see collinearity as being more a problem of *identifiability*: when variables are almost collinear, each becomes almost interchangeable with linear combinations of the others, whence any interactions also become interchangeable with linear combinations of interactions.
@gmacfarlane thanks for the welcome! like I have mentioned in my post above, I have looked up at `DPpackage` and also at `GPfit` package of R but I am unable to understand which method inside these packages are suitable for my purpose and how to process my training data and testing data for that
"have outliers" already implies "not Gaussian"
Nope, that's it. It is perfectly acceptable to use Booleans as predictors / covariates in a multiple regression model. It is very commonly done.
The dataset I plan to upload to dropbox shortly as an .RData file. Simply download the file from dropbox and when you open R, use the command `load(file_location)`, and then use `head(df.add)` to confirm the loading of the dataset. The x values are the start column and the y values are the Number_of_AID_OH column. Ignore all other columns.
+1 nice links. Thanks.
After doing the overall test picking a pair and saying that that pair id significant interms of differences in the center really amounts to doing the pairwise testing without actually going through teh multiple comparsons.
This answer does not actually address the OP's question about difference in significance between the two statistics.
@ Dr Jochen L. Leidner -its considered good form to post some details of any links you supply, so that the answer can remain useful even if the link is broken. Welcome to the site
This answer assumes an specific interest in learning regression from an econometric point of view. That interest is not explicit in the question. It is naturally true that econometrics texts are very strong on regression (and in some cases authors even appear to know of no other statistical method).
-1; see @Bernd. Most mature packages use compiled code. The main reason is actually that BUGS et al are more flexible in that they can fit more models. While this may lead to more efficient computation since an R package - even with compiled code - has to be more general, it may not.
Look at Statistics Topics ebook on Amazon by Mehta, and his free web log Statistics Ideas that has lecture slides. Nearly free and better in some pedagogical topics, than the ones you cite on your list of resources.
Dear Jake, thank you for you comment, it's very helpful. I got it: indeed, the questions being asked are different, and so can be the answers. Beautiful!
Are you doing spearate tests for individual months?  Is the correlation only across months with the cases within a given month independent of each other?
@gung I think there might not be enough repeated measures (9) for a random effect of time. Maybe a fixed effects model is better.
Could you clarify what you mean by "account for the mean squared error?" Using the transformed variables, lm in R will produce the correct variance on the transformed scale. But do you want to know how to back-transform the variance to the original scale?
Great that really clears things up. In some of the resources i'm looking at on the subject it seems like it's making a distinction.. that being a "Method 1" where you use a uniform distribution and a "method 2" where you use just about any other well fitting pdf. The distinction was throwing me off but it seems that either one will yield samples. It's just a matter of efficiency/acceptance rate
I think that is right.
@SolLago: Edited the answer to address your question and deleted my comments that lead you down the wrong path.
[this site](http://www.multiple-imputation.com/) might give answer to your question. The link `What is MI` gives a list of various resources.
In case you're still interested, the outcome of rolling a 20-sided die is uniform over all faces of the die, so the probability of observing any individual face is $\\frac{1}{20}$. However, the number of heads on 20 coin flips follows a binomial distribution where the probability of $k$ heads is $\\binom{20}{k}\\left(\\frac{1}{2}\\right)^{20}$.
Variables such as holiday or working day can be included by transforming them into dummy variables, for example in one week there is $(1,1,1,1,1,0,0)$ where $1$ is a work day and $0$ is holiday, and you can see the difference on the coefficient of the dummy. If there are variables with several ordinal categories, you should considered using a ordinal logistic regression.
This is true only if we assume that *your friends* are representative of the overall group.
Very closely related: http://stats.stackexchange.com/questions/29121/intuitive-explanation-of-unit-root.  Perhaps it's the same question?
Are your variables continuous? CA is for categorical variables. If you just want to plot them in a lower dimensional space, you could possibly use Gower's distance to combine continuous & categorical data into a distance & use multi-dimensional scaling to determine the dimensionality & plot.
yes, fitting crossed random effect is not issue alone however fitting user defined co-variance structure is main issue and the question primarily seeks answer for that.
I agree with Stask on Lohr's book.
@styfle - the PDF can also be seen as the probability that a RV lies in an infinitesimal interval.  $Pr(x<X<x+dx)=F(x+dx)-F(x)\\to \\frac{dF(x)}{dx}dx=f(x)dx$ as $dx\\to 0$.  This way holds more generally, even for discrete RV and RV without a density (the limit is just something other than a derivative)
you are absolutely right. I was looking at surv=0 and not surv=1. Thanks for the help!
Sure: but exactly what supposition are you making about the departure from iid?  There are many, many ways that can occur.
You should use 1.645 when the population standard deviation is known, if not (common situation) then use 1.706 value. There is no explicit formula to calculate $t$ values, however there are [tables](http://en.wikipedia.org/wiki/Student%27s_t-distribution#Table_of_selected_values). Why not using build-in functions?
Just to be clear, you're saying that it does converge, but it takes an unusually large number of iterations, right?
How much does this decrease the power of the analysis? In what conditions is it most problematic? 
In many cases deviations from the assumptions of ANOVA do not substantially affect the outcomes to an important extent.
@Chase, when I commented, there was no R code example.
For clarity - you are simply asking about how to *present* the results of the accuracy in a table? If that is the case, then I answered a similar question on the [Academia site](http://academia.stackexchange.com/a/23142/3) and I would give the same answer here. The Feiberg and Wainer citation suggests to try to limit significant digits to only three numbers in tables.
Someone with enough privilege, or martin himself, should add the tag regression, please.
Are you sure you don't want to consider replacing the 100.00% too?
Excellent, thank you Nathaniel!
No, it's some practice questions I found on the net for Msc Statistics. 
1. Oh right, I got my tree incorrect. but calculations correct. made the fail rate + 10% each year instead (ugh)
2. ok, I failed on basic circle theory ... I did 2^2π for the second calculation. 
<br/>Thanks a lot for clearing my mistakes!
@Aaron "Narrow inference" is discussed in this SAS documentation: http://www.math.wpi.edu/saspdf/stat/chap41.pdf But I haven't tried to understand yet.
I think it should also say that from a practical point of view the population variance can never be estimated.
This belongs on CrossValidated. I have flagged for moderator attention and the question will probably be migrated soon.