tttjjj/elt_bench huggingface dataset:
Followup - Case 2 had k_21 successes in n_21 trials in one month, then B,C and D were applied to Case 2, and then it had k_22 successes and n_22 trials in the following month.
If you condition on the observed count, $n$, the expected number in each cell is the probability of being in the cell times $n$.
When you say non-stationary variable, what kind of non-stationarity do you have in mind?
alright, how about this:  http://www.creativedistraction.com/demos/gesture-recognition-kinect-with-hidden-markov-models-hmms/  it's a step-by-step demo with code of how to build your own gesture recognizer with the kinect.  code is for Octave not python, but it seems like you are stuck on the ideas/implementation more than the particulars of the software.  of course it requires a kinect...  you might also check this out:  http://code.google.com/p/ehci/
You're not classifying data as outliers on that basis alone, are you? Why do you think this would affect the matter of reporting significance?
Excel can compute inter-column correlations but I don't know if there is any clustering functionality.  The usual method to do anything complicated with a spreadsheet is to export it to a CSV file that can be processed by statistical software such as **R**.
Yes that is what I was looking for..I couldn't find it anywhere  better explained. Thanks.
You technically need quite a bit of level-3 units if you're going to get unbiased parameter estimates in a 3-level model (generally speaking, sample size in any multilevel model is particularly important at the highest level), so unless you have a large random sample of countries (50+ perhaps), you should probably consider running separate 2-level models, or if you have few countries, you could consider treating country as a categorical level-2 predictor in a 2-level model
To sample y, just sample x from a Normal(0, 0.2) distribution and compute y = e^x/(e^x+1).  You can use (0, 0.2) as the parameters: by means of this formula they completely determine the distribution.
@Bitwise Yes. I believe that the problem will be small enough that an exhaustive search would in fact be okay for my application, but I'm still a bit lost on how to approach it. I think it will be a problem with perhaps 20 options and the total sum might be 25.
My answer here: [how-do-i-compare-bootstrapped-regression-slopes](http://stats.stackexchange.com/questions/18707//18726#18726), may help you. For a CI on the slope, #1 is the way to go. It is fine that some bootsamples are not significant. "The difference between significant and not significant is not itself statistically significant"  -Gelman.
Did you scatter-plot the data? It shows curious shape
I'm sorry if this site isn't a good match for this question; if not, can it be migrated to the appropriate site instead of being closed?
*I can't get it to work for some reason* -- are you blocking javascript or something like that?
I was thinking of confidence intervals, but the question is not clear to me because there seems to be different outcomes each time (but may I don't understand the question) which would prevent from pooling anything at all.
This is a very overloaded term, depending on context. You have been warned.
The Jane Miller is book is really quite excellent.
Yes, reference value is the same as baseline value. You will notice in the R equation output it says `reflevel = "0"`. My baseline is 0. Yes the mlogit package requires the response variable to be a factor. I did correlation analysis of all my variables and none are above 45% (except age and age^2 but that tests for nonlinearity).
@StasK, your statement RE: Excel is false. I don't have it on my computer and it costs $$$.
@chl I used [JMP](http://jmp.com) on a Mac, a product I help develop.
Hi Naive Prestige; welcome to CV. I edited your question because the first and second parts didn't correspond. I plumped for adding 'not' in the second sentence. If that's wrong please edit your question to remove that and correct the apparent mismatch between the two sentences
Your statement beginning "In the underdetermined case..." isn't quite right, as the minimum norm $\\beta = 0$.  It's the minimum norm $\\beta$ given that the norm of the residuals $= 0$.  Also note that in this case $(X^TX)^{-1}$ won't exist, as $X^TX$ won't be of full rank, so your question, at least as stated, has no answer as you can't in fact solve for $\\beta$ in an underdetermined system using the left inverse.
Dies the "Q" stand for "quantum" here, or is it genre jargon?
Thanks very much - very helpful. I will investigate `anova.coxph` too.
That's so. Procrustes is one (and the simplest) way to go. Confirmatory FA is another. On my web-page there is an abundant SPSS macro doing procrustes rotations for two groups.
Hi Hanna, my suggestion is Applied Longitudinal Analysis by Garrett Fitzmaurice, Nan Laird & James Ware http://www.hsph.harvard.edu/fitzmaur/ala/ - it's a good introductory book with tons of worked examples.
By using only the returns you become very vulnerable to ladders (i.e. a sequence of prices that climbs or drops away from the norm, where each individual return is acceptable, but as a group they represent an outlier). Ideally you'd use both the return and the absolute level.
The notation in your previous comment makes no sense: the value of a *realization,* such as "$0.5$", cannot have an expectation! You seem to be confusing the realizations with the random values used to model them. The value $0.5$ is a realization of a Uniform$[0,1]$ variable; likewise, $0.75$ is a realization of another Uniform$[0,1]$ variable. The *variables*--which are measurable real-valued functions defined on a sample space--have expectations, while their realizations are just numbers (which we anticipate will change in repetitions of the experiment).
Thats it. Perfect. Thanks.
1) Lme4 doesn't give you p-values, because for unbalanced, multi-level data, denominator degrees of freedom are unknown. Most packages give an approximation. Douglas Bates (lme4 author) instead chooses not to. More information found [here](https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html)
2) Nested effects really depend on the structure of the data, so it's hard for me to see what you want to specify from the example. [Here](http://stats.stackexchange.com/questions/13166/rs-lmer-cheat-sheet) would be a good place to start
Tried to using the method says in the book of Elements of Statistical Learning, but don't know how to create the basis matrix  in this case.
(1) If the values are the same the variance of the $x$ or $y$ would be zero. The correlation bit was where I got confused with the multivariate case, sorry. Check if either variance is close to zero.
Hi momeara, thanks for that info. You're suggesting I do what seems to be a fairly sophisticated analysis (for my level of expertise). I take it that this means that it's inappropriate to use a more straightforward approach (e.g. Running a "standard"  GLMM using lme4 in R) with only 3 observations within each level of the random effect? Cheers
Poisson regression is not such a great idea for binary data, and is a very bad idea if the binary data is not numerical (as is almost always the case, e.g. "Did you vote for candidate $x$?"). Not to mention, under the model, non-zero mass would be given to points other than `0` and `1`. Echoing @PeterFlom - why would you want to do this? Is something wrong with using standard techniques for binary data (e.g. logistic regression)?
Bootstrapping involves training a large number of models on resampled datasets, where the resampling is performed with replacement, and then use the samples in each iteration that are not used in training to test the model (so it is a bit like repeated randomised cross-validation).  Bagging means using the models generated via bootstrap to form a committee model, see http://en.wikipedia.org/wiki/Bootstrap_aggregating .  The repeated retraining can take a lot of time, but it is a good belt-and-braces approach to avoiding over-fitting.
In a way, I think I am interested in the shapes of the graphs. For example in my data I have graphs in which new nodes are cumulatively added. There are also graphs that start with many nodes yet only one node makes it to the end.  I would consider these two types to be structural opposites.  Graph theory is really not my field, so maybe you can help me with formulating my question more specific?
This is a question about selection. It is a common topic here & elsewhere. You should *not* only include significant variables, to understand why, you may want to read this: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836/20856#20856). You should also search / read around CV under the tags: [tag:feature-selection], [tag:model-selection], & [tag:stepwise-regression], for starters. If there is still something that you want to know after having read through that, edit your Q here to clarify, otherwise this Q should be closed as a FAQ.
Thank you, that cleared things up for me.
The objective function is $U(\\beta)=\\sum_i (Y_i-\\hat{Y_i}(\\beta))^{2}$  We differentiate w.r.t each beta $\\frac{\\partial U}{\\partial \\beta_j}=-2\\sum_i X_{ij}(Y_i-\\hat{Y_i}(\\beta))$.   We set to zero, so I can multiply the jth equation by $\\beta_j$ (still zero on RHS), and I can add together all of the $p+1$ equations (still zero on RHS), and we have $-2\\sum_j\\beta_j\\sum_i X_{ij}(Y_i-\\hat{Y_i}(\\beta))=0$. interchange sums $-2\\sum_i (Y_i-\\hat{Y_i}(\\beta))\\sum_j\\beta_j X_{ij}=0$, divide by $-2$ and we have what I said as the result, $\\sum_i (Y_i-\\hat{Y_i}(\\beta))\\hat{Y_i}(\\beta)=0$.
@Anony-Mousse I think that's the reason I am using cross-validation  (or test data) error rate. So I think I have less possibility to fall in that category.
1) What is `R` or `delta R`? 2) What do you mean under `common factors` when speaking of PCA?
cluster classification can add new information to the regrssion, I think. If you run, say, a random forest to classify but a linear regression to regress, then clearly the random forest could split the data in non-linear ways that are precluded to the linear regression. Am I making sense here?
Thanks Simone for the note on Random Ferns. You are right, 1,000 is a lot of classes. In my usecase, I'd like to use something in the order of 100 classes.
If you are only interested in prediction, then model selection doesn't help and usually hurts (as opposed to a quadratic penalty = L2 norm = ridge regression with no variable selection).  LASSO pays a price in predictive discrimination for trying to do variable selection.
Thank you for helping to clarify. Raw outputs may be too difficult to compare due to the sheer volume of data (100+ participants, tracked every min for 10 days). Users wear the devices differently and they measure alternate criteria (both contain an accelerometer). I'm weary of performing a correlation for the aforementioned reasons....maybe I could do one for PAL (i.e. total energy expenditure/basal metabolic rate) and then use Bland-Altman plots for the more in-depth 'bout intensity' analysis. I will continue looking for a paper that compares two devices in a free-living setting without DLW
independent random variables ?
Note also that neural networks are generally not convex, which means that the final result can depend strongly on the initial conditions. In the Matlab neural network toolbox, the initial conditions are chosen randomly, so your results may be strongly influenced by random chance.
Is a model really necessary? Why not simply count which paths or which transitions are the most common?
For a reference on the potential for different types of error bars, see Using Confidence Intervals for Graphically Based Data Interpretation ([Masson & Loftus, 2003](http://www.unt.edu/rss/class/mike/5030/articles/MassonLoftusCI.pdf)). Also pertinent, see some critiques of Dynamite plots, (see [Drummond & Vowler](http://dx.doi.org/10.1113/jphysiol.2011.205062). 2011. Show the data, don’t conceal them). Also I give many references to discussion about error bars (and examples of how to display them in ggplot2 in R) in this [post](http://goo.gl/qXjY8).
(+1) It might help to be more specific about what you mean by "compare the residual sum of squares" (such as the test statistic and it's null distribution).
@Affine Are paired t-tests most appropriate here? My reasoning is that the dataset is the same for each algorithm so is probably a repeated measure.
@ Peter Flom And then? E.g. for the 95% prediction interval, the rule of thumb is fitted value +/- 2*SE. Then the better model is the one with narrower interval, right?
Because you have only 2 classes, if it fails to classify to this class, then it means it wins to classify to the other class.
If you are using PCs from a PCA within some other procedure, their origin in PCA is immaterial for significance testing with that other procedure. That is a little contentious, as statistical people don't all agree on whether PCA is a multivariate transformation procedure or model estimation, but I think it is a good first approximation. If that argument is accepted, then your question is just about significance testing in whatever you are doing and covered by any standard account. Do  you regard linear regression and PLS as equivalent? You seem unclear which you are using.
Andrew Gelman has a set of unpublished recommendations about standardizing categorical variables [here](http://www.stat.columbia.edu/~gelman/research/unpublished/standardizing.pdf).  I may write up a longer answer if I have time.
@David I want to have some credible interval for the predictions (`pri`), this is why having `mu` is not enough. If I append `pvr` and `pir` I would get `alpha`, `b.vr` and `b.ir` for the whole `ri`, right? Or can I have `ri` with missing values e.g. (1,2,3,NA,NA) and append the regressors?
"[T]hen I am making very strong assumptions about the relative difference between consecutive values of the ordinal variable."

I think this is the key point, really.  i.e. how strongly can you argue that the difference between groups 1 and 2 is comparable to that between 2 and 3?
I have edited the question. Please see the last paragraph. An no, this is no homework nor assignment.
@user862: I believe stationarity's intuitive meaning is that the data looks the same at any point in time: if someone hands you two graphs of the data at different points in time -- say with the x-axis blacked out -- you won't be able to tell which one came first.
It's continuos. The data is basically words with sentiment scores (in the range of -5 to 5).
Monty Hall is one of those "counterintuitive" things, which is why people have problems understanding it. [Here is a video](http://www.youtube.com/watch?v=mhlc7peGlGg&feature=plcp) that explains it in a simple way. Plus, the __MythBusters__ have [tested it](http://mythbustersresults.com/wheel-of-mythfortune) and gave it a _Confirmed_.
Do you mean that you have 6 values that are out of line? Can you post the data?
And probably not easily put on an iPod, if that is essential to you.
Maximum likelihood for a normal is least squares, but least squares is BLUE (via Gauss-Markov), so you don't have to assume normality to want to use it.
I'm not sure I understand what, exactly, you mean by 'significantly separated in time'. What would be the null and alternative hypotheses that would lead to a decision about statistical significance?
I think you should start with examining how the disagreement between the sensors depend on the temperature. Plot the difference between the two readings against the sum of them.
@gui11aume, thanks for you input! However, your statement "p-values are a relatively late addition to the theory of statistics" is strange. From what I have read, Fisher's 'significance testing' with p-values originated around 1925. While the Neyman-Pearson's 'hypothesis testing' came about as an 'improvement' to Fisher's work a few years later. While it is true that p-values were difficult to compute (hence why standard levels of significance were used), his work was monumental. In fact, he is referred to as 'the father of statistics' because he formed the basis of much of modern statistics.
It also depends on whether you would like to attempt to interpret the character data and translate it into a number, or just treat it as missing, in which case Zach's answer below does the trick.
I actually want to use the correlation matrix found from about 40 variables like the ones I mentioned above, in the factanal() function in R to make a factor analysis. I want to make comments about how much each factor affects profitability. So I wanted correlation matrix from ordinal variables and mixed type variables. Thanks all for your kind suggestions.
I think there is a confusion of concepts about "bias" here.  Bias is a quality of a statistic, which is an estimate of a parameter.  You can only talk about a statistic being biased if it is an attempt to estimate a parameter and can be shown to on average get it wrong.  I don't see what parameter your statistic is trying to estimate.
@rocinante I've edited my answer to clarify that in the scheme I gave they're not estimates of different things, they're actually estimates of the same thing ($\\beta=\\frac{_{E(Y|x)}}{^x}$) under different variance assumptions. If any of the x's are zero, both the OP's estimators are sub-optimal, but still give consistent estimates of $\\beta$.
If applicable, see the [self-study tag wiki](http://stats.stackexchange.com/tags/self-study/info). (I'm guessing you're not just asking out of curiosity, but if you are, more power to you!)
Another reference: http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1263478373
${2m \\choose {k_1,k_2,...,k_n}}$ is the number of ways 2m distinguishable working units can be partitioned into n slots with exactly $k_i, i=1,...,n$ in each of the slots. Now we need $k_i \\geq 0$ since they represent the number of units in a slot. We also need $k_i \\leq N$ since we want the "overlap" less than $N$. Also, since we have $2m$ distinguishable working units, we need the sum of these counts $k_i$ to be equal to $2m$. Any such n-tuple of $k_i$ is a valid solution. We are summing over all of these.
+1 – We must test this hypothesis immediately! I've always enjoyed fresh fruit, so I'll volunteer to provide an observation at $a=o=10$. Someone give me 30 apples and 30 oranges; I will report back in a few days.
@whuber: i was hoping that some clever chap, such as you, would have some insights about how to approach this. btw - there is a longish list of related problems on the RHS of the screen, to many of which your question could be applied.
Thanks, I followed your advice on Levene and the odds ratio. The problem may not appear when prob=1 for the reasons you mentioned but there are cases when it is zero.For this I just replaced 0 and 1 with 0.001 and 0.999 and hope for the best. We can not be sure about the distribution of lengths, they appear normal but my suspicion is that other quantities (e.g. weight) consist of subpopulations (e.g. let's say those cuc inseminated [!] by a bee in the morning and those done late in the afternoon etc. etc.). So, correct me if wrong, but may need to do gaus mixture and then apply same procedure.
Compare is pretty vague. Do you want to test equality of effects of explanatory variables on the latent or actual outcome? Do you want to test if the tobit is appropriate (normality, homoskedasticity, same DGP for zero and above zero)? Do you want to see which model fits the data better?
It's hard to tell from the graphs, but are there a whole lot of 0s?
See the help center (e.g. http://stats.stackexchange.com/help/on-topic) for more advice on code-related questions.
I really like your 2nd paragraph.  But in the 1st I think you make the mistake of concluding that subtracting a constant from every value will change the shape of the distrubution.
Any chance I can send you what I have now so you can take a look?
@Tomas `lmer()` *doesn't* use REML when you ask it do comparisons. IIRC they included some fancy sugar in `lmer()` so you didn't have to refit the model with `ML` just to compare fits when you want `REML` on individual fits to get best estimates of the variance parameters. Look at `?lmer`, run the first LME example up to and including the `anova(fm1, fm2)` call. Look at the log likelihoods reported by `anova()` and those reported earlier in the printed output. The `anova()` is getting ML estimates for you.
Hint: being more concise will probably help you get an answer.
@JDL, If were to claim that I am taller than everyone in your family and we wanted to test that claim then we could either compare my height to the height of each member of your family, or simpler would be to just find the tallest person in your family then compare my height to theirs.  It is similar with the 1 sided test, we use the equality for the null distribution and if we reject that value (showing that we believe the true value to be less) then we have also rejected every value greater than that null value.
I must say that I've seen plenty of people in the industry use Bayesian methods for computing conversion rates, and I also must say that I don't know what the value of $\\kappa$ is, and I don't trust the maximum likelihood estimate of its expectation if I don't have that much data. Of course, if I do have lots of data, the two approaches give the same answer, so who cares? Well, I do, because it turns out at least for me that the Bayesian approach is easier to understand. To each their own I guess.
Look at Geoff Hinton's and Andrew Ng's qualitative youtube lectures for an easy overview of why it's so good.
I'd insert "not" in one sentence you quote "In statistics, the expression above is often referred to as the normalized correlation coefficient", i.e. "not often". But the explanation you cite seems fine, so far as it goes. This is just a standard correlation. If the name is qualified ever, it is as the Pearson product-moment correlation. I'd say that "product-moment" was a quite common label decades ago, but has faded in use, while "Pearson" remains common. But this is all, and only, about terminology.
@Aksakal: The exact time is more important than the exact volume. The data represent shipments and it is important to know when they are expecting to happen so plans can be made accordingly.
"Diminishing" will not take you to N^2. I don't think this is the optimization done.. regarding your second comment, Im planning on calculating the thd on a small data, and setting it on big/large scale data. My output will be (hopefully, for my needs) 99% singlton clusters and only a very very small amount of clusters containing more then one sample in it.
Nice answer! The chorus of Phil Ochs' song [Draft Dodger Rag](http://www.cowboylyrics.com/lyrics/ochs-phil/draft-dodger-rag-11443.html) lists possible mechanisms to avoid the draft. I always thought checking these would have made for a fun paper.
@cardinal - Yes, thanks.  I appreciate your comments and have revised my answer so that it hardly resembles my original answer.
Classification can be needed if the ultimate decisions need to be made instantly with no user input regarding utilities (loss/cost function).  Is that your setting?  Otherwise a risk model may be more helpful.  For one thing, risk estimates provide a "grey zone".
@KHKim You're not bothering me. You just need to register your account (see the [login page](http://stats.stackexchange.com/users/login)). Then you'll be able to get system-wide notifications, leave comment and edit your questions, etc. See [Do I have to log in or create an account?](http://stats.stackexchange.com/faq#login) in our [FAQ](http://stats.stackexchange.com/faq).
Why not use a continuous approach that has much better statistical operating characteristics?