tttjjj/elt_bench huggingface dataset:
The paper you link to seems to answer your questions. Perhaps you can rephrase your question to ask something specific in the paper that you did not understand?
+1 It's always nice to see such useful historical material.  Although it's only implicit in this quotation, the appeal for Ulam, von Neumann, and others is that this technique promises to reveal *complex* behaviors of systems, such as their dynamics and their natural variation, which goes far beyond mere estimations of expectation as in the Canfield game.
Different number of variables in the 2 sets is not an issue, it is normal. If you have missing data within the sets, try to imput them, e.g. via EM algorithm.
If you are familiar with R, there is a package `vegan` with functions `anosim` or preferably, `adonis` (permutational MANOVA).
I'd imagine that this has implications for power/estimation accuracy (i.e. follow up was not as long as intended for participants: and may actually be too short to be of use full-stop if e.g. study ends at 6 months for a process where events would be expected mostly at 12 months +) but from an analytical point of view I don't see any difference between this kind of censoring and the kind that would happen when follow-up reached its planned timing.
@KateRin you mentioned that this is a result of some function. Based on the output that I see, I don't see any major residuals at all. Dis you add a random noise to the data from the function, if not then you cannot expect any residuals. As mentioned it is a somewhat a good fit. The code that I provided is a very basic code you can improve it to get more robust fit for your data.
(1) There is something wrong with your formula. What $a_i$ might be and why both expressions end up at $p$? (2) The first canonical variate pair is U1,V1. Correlation between them is the 1st canonical correlation.
-1, a bit off target:  The Z-statistic is normal when the entire population has been sampled to find the population mean and standard deviation. Which I think I acknowledged in the question.  The accepted answer (Greg Snow's) provides the sort of answer I was looking for.  That is, an answer that talked about the insufficiency of Student's estimation when large proportions of the population are sampled.
It will be much easier to explain this if you provide more details of both models (in particular: provide both models with a description of the variables at hand)
@ttnphns I saw a statement that one should never do a Pearson correlation two variables measured as proportions. I've tried to make this clearer by editing the OP to highlight the word 'never'. The video makes the same statement in its title ("Don't correlate proportions!"), though they only discuss this in the context of compositional data. I deliberately left the context undefined because my source stated that Pearson correlations should not be used on proportion data in any context. However, it seems the answer to my question is: "Correlating proportions is fine, except in some contexts."
@DaveM, it's one way to de-trend if you know that the time trend is linear. what if it's not? de-trending is often done in the context of [time series decomposition](http://en.wikipedia.org/wiki/Decomposition_of_time_series)
Because this is one of the best approaches imaginable--you only have to decompose the 3 by 3 correlation matrix once and linear combinations of simulated random fields are fast and easy to compute--why do you seek a different algorithm? What is the underlying objective to finding an alternative?
I think that's too strong a statement in at least two senses. 1. In principle, Pearson correlation could be applied to two counted variables, for example. It is perhaps unlikely that would often be useful, but that's not the main point. If linearity of relationship makes sense, Pearson correlation can be applied. 2. Chi-square testing is a weak form of analysis if one or both variables is discrete but ordinal.
@MichaelMcGowan I wouldn't argue that point with you.  i generally distinguish multicollinearity from overfitting the way you do. I got caught up in explaining what happens when too many variables are in the model and combined them under the umbrella of overfitting.  That might have made it easier to explain to the OP.  But I didn't like doing it.  Apart from the semantics we do agree that mutlicollinearity does not necessarily hurt prediction but overfitting does.
try me, I know Russian, and formulas are the same for all languages.
@JandR: You're not being slow. We're just trying to figure out the real question of interest. If a sequence of random variables converges to a degenerate distribution, then the variance of the limiting distribution will be zero. However, I believe the variances of the random variables in the sequence need not even be defined. Take, for example, a sequence of Cauchy distributions with a scale parameter converging to zero.
The categorical variable looks very suspicious, not because it's missing "J", but because the whiskers do not overlap at all, which prompted me to think that the categorical one is just a regrouped version of the continuous one. If that's the case, it's quite unlikely that both of them will be retained, because the continuous one is likely going to knock the categorical one out: the categorical is less predictive to begin with, and it takes whopping 11 degrees of freedom (12 if the blank one is real). If they are different things, then as the comments above said, we will need to know more.
@Greg I don't think there is much to worry about using ad hoc cutoffs.  That is typically what is done with the chi square tests.  The specific binning doesn't effect the test unless you make it far too narrow or too wide and good judgement prevents that.
What specifically is your "model"?  What is the form of the log likelihood?
Add data and graphs. Otherwise it's difficult to know what you mean. Also, what does "better in terms of confusion matrix" mean? You can order them by accuracy, or f-score, or precision, or recall, or sensitivity, or specificity, ....
Before I update the answer, is treatment a repeated factor? i.e., does each subject receive both treatment "a" and treatment "b" or is this a between-subjects factor?
Yes, you could use some Fourier terms as regressors and deal with the seasonality that way.
A proof of what?
Thank you. Help much appreciated!
Hi Luna, if you've found either of the answers here helpful, I hope you'll  consider accepting one of them. Also, if I can clarify anything in my answer, please let me know!
You do not need a "mixed effects logistic regression model" to estimate a simple discrete-time model (Fiona Steel has published a few articles on "[Multilevel discrete-time event history analysis](http://www.bris.ac.uk/education/people/fiona-a-steele/pub/2781928)"). Do you have a reference? Re the data preparation step, I also suggest to have a look at the [survSplit](http://stat.ethz.ch/R-manual/R-patched/library/survival/html/survSplit.html) function.
Do you wish for more information?
Since it's ARMA, there's an issue with dependence across the change date.
An even better solution is to directly use `summary.formula`, e.g. `print(summary(~ x + y), prmsd=TRUE, digits=1)`. The `latex` command assumes the OP has a working $\\TeX$ installation. It is often convenient to just output plain text, like this: `capture.output(print(summary(~ x + y), prmsd=TRUE, digits=1), file="out.txt")`.
Even if it is not off-topic, it should be community wiki.
I altered the question to try to make it general and useful for others who may read the question.
To scale element $e \\in [a,b]$ to an interval $[c,d]$ you have to calculate $\\frac{e-a}{b-a} \\cdot (d-c)+c$.
@user28327 If that answers your question, you could “accept” the answer by clicking the little check mark under the question's score on the left.
Thanks.  This was essentially just what I was looking for.  (I should have googled crowd size instead of population size, yours is there with a bunch of others).
@Seb Thanks for that additional information!
As the score is the proportion of variance explained by this factor. but it doesn't have two be related to the discrimination property. isn't it ?  (sorry for the not so good English)
You should definately make your question self contained and more clear to get better/any answers.
This is not actually simulation under the assumptions of regression, but bootstrapping (resampling your data). Why do you think this will have the right coverage probability?
Actually yes, currently the diagnosis is a composite score, along with biopsies & other measures. However I intend to build the classifier without using the tests and just using the biopsy/other measures to determine the class label, to avoid circularity
I agree with gui11aume.  Also when reaults are so different it may indicate different assumptions or some other subtlety that makes the studies different and not proper to combine.  It could be looked at a little like the old joke about statistics.  Man puts one foot in a bucket on fire and the other in a ice bucket.  On average the temperature is normal.  You wouldn't want to describe this with the sample mean. It is the same with the meta analysis averaging results from two studies with widely different results.
@Gaetan-thank you.
Are you more interested in how well the model will eventually perform for classification (in which case ROC and AUC type of analysis seems most relevant  (http://en.wikipedia.org/wiki/Receiver_operating_characteristic)?  Or are you more interested in understanding how "calibrated" the probability predictions are (i.e. does P(Outcome = A)=60% really mean 60%, or just that outcome = A is more likely than the other outcomes...
@Nick Cox:  probably.  I'll optimize the code once I get it functional.  I started trying to make a function to demean a whole data frame, handle missing data, etc, but pared back my goals when I realized that I wans't getting the correct results in the first place.
The algorithm is not the same: 
`Fstat` tries to find one break point (i.e., two segments).
`breakpoints` tries to find the optimal number of segments,
basically by trying all possible breakpoints 
(the manual page `?breakpoints` gives more details about the algorithm).
`plot(b)` shows how the number of is chosen
(the algorithm tries to minimize the BIC).
What you want is probably a plot similar to `plot(f)`,
but not limited to a single breakpoint: I am not sure there is one.
`breakfactor(f)` issues a warning: it should be `breakfactor(b)`, 
which returns the 3 segments.
It's a 6 month forecast, so after the first month pass by the observed value (what occurred) will not be appended to the data. That's the problem: i need seasonality to take care of these variations that you mentioned.
What does PPC stand for?
@Harpreet When constructing confidence intervals, say a 95% CI for a difference of means, you are explicitely assuming a sampling distribution (t or z, i.e. that of your test statistic under $H_0$).
Feel free to accept the answer or to comment on what you are missing from it :-)
+1, this is what I was trying to express but was not able to :)
This is a tough problem because you have very little information about the nature of the cheating. How do you differentiate a cheater from a student who studied extra hard? Without more information, you can't. One possibility is if students can cheat by copying off each other, or if subsets of students had access to the same answers. If this is the case, you could create a distance function between students (lower distance means they did well on the same questions) and look for patterns here. This would be more conclusive IMO.
It now sounds that you are using "size" to mean "count."  You cannot even begin calculating correlation on such data: they have to be paired.
Thanks for your helpful reply, Dr. Harrell.  I'd just like to sum up my understanding of what you said.  Please correct me if I'm misreading it:

 
`fit.mult.impute()`: Use this to pool the `cph()` models (5 of them, based on 5 imputations from `mice`) and obtain pooled hazard ratios and standard errors. 

`transcan()`: Use this to create a single imputation and validate that.  It sounds like this gives a good enough validation.  

Is all of that correct?  I really appreciate your help, Dr. Harrell.
Very cool, I was going to suggest not an entirely dissimlar type of sparkline. They weren't orignaly meant for time series, but the idea can certainly be applied to it. The seperation plot ([Greenhill et al., 2011](http://dx.doi.org/10.1111/j.1540-5907.2011.00525.x) - pre-print PDF [here](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1450589)) is a visual technique for binary data that can be displayed in a very small space (i.e. sparklines).
I think the issue is that it is "difficult to distinguish real peaks from random noise" because of "false" peaks such as $(0.02, 0.50, 0.12, 0.09, 5.02, 0.88, 0.01, 0.23, 0.11)$ as opposed to what Henry wants to detect, "...a classic peak shape with the values slowly rising to a peak...," such as $(0.50, 1.10, 2.33, 4.44, 5.02, 4.01, 1.87, 0.94, 0.01)$. So $T_i(x)$ needs to be chosen not as the value at the center point of the profile $x$, but instead as a function that captures whether there is a pattern of increasing values before the peak of $x$ (and decreasing values after the peak).
It might be ideal if you could add a few lines about Chisini's definition here in case the link goes dead, &/or to help readers know if they want to click the link to pursue the ideas further.
Also, the multinomial probit model gives comparable output with a different set of assumptions. So an MNP/MNL comparison can be valuable as well.
Why the difficulty? $F_1$ should not be a subset of  $F_2$ and $F_2$ should not be a subset of $F_1$. So form all possible sigma-algebras permitted by $\\Omega$, and pick two that satisfy this.
In practice, however, the autocorrelations you show are all very small: although some are significant at conventional levels they should not be over-interpreted. Correlations about 0.02 don't imply much predictive ability.
Thank you so much!  That was exactly the problem.  The first line of the file (the non-numeric line that specifies the names of the fields) had accidentally been copied several times in my latest run; removing the extraneous copies let R realize the field was really numeric.
@Elvis please read the the last item on the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) on the `self-study` tag; it explicitly asks that you don't add the tag yourself, but check with the OP. Hence my previous question to the OP. (Where the OP has clearly indicated that it is self-study but hasn't added the tag, it wold be reasonable to add it.)
I use a linear kernel. I assumed that you don't need to tune any hyperparameters in relevance vector machine, since these are 'tuned' iteratively already.
I've never heard of "normalized correlation", and cosine is a better word. Correlation itself is the cosine after centering of variables ([see](http://stats.stackexchange.com/a/22520/3277)).
A more explicit answer below. I'll give an outline of one way to do it.
What do you mean by a Gaussian with a variance *and* scale parameter? The variance parameter *is* the scale parameter of a Gaussian! I'm also a little unsure about the model you've set up so far. Can you describe the problem you're actually trying to solve in more detail? Using a Gaussian to model low granularity integer-valued observations seems fishy.
For question 1: in `?lme` it gives: `correlation	
an optional corStruct object describing the within-group correlation structure. ... Defaults to NULL, corresponding to no within-group correlations.`    So default is uncorrelated repeated measurements.
Depends on what makes sense and what you want to show/know about your data; you can try subtracting mean (column-wise, row-wise, global), show differences or quotients, also in function of one or other variable, subtract this data from one (and maybe use log scale), draw scatterplot...
note that this will be your accepted answer for the bounty (in case you do not choose an answer the bounty automatically select the answer with the most points)
Can you explain me how the algorithm works? The median is that value such that the integral of the probability density function is equal to 0.5 ? Is the median defined for a histogram?
What about shifting to negative binomial or use Poisson regression with robust variance estimates? Both are supported by SPSS.
Thanks. I have uploaded a sample file at https://dl.dropboxusercontent.com/u/15643073/Example_Data.csv . The file contains Columns by Quarter for POP (Poulation), INC (Income) and Sales. The final few rows have all the data except the Sales for Q4. I generally use R for most of my analysis. Thanks a lot for your suggestions on how to address this issue.
Given this information, you can't do this.
"*It simply isn't possible to construct a confidence interval for a binomial proportion which always has coverage of exactly 95% due to the discrete nature of the outcome*" --- aside, perhaps, for the (somewhat odd) possibility of randomized intervals. (At least in that way, it *can* be done, though it may well be that it usually *shouldn't*.)
@Rasmus Bååth: I am using R, and, to be honest, I do not have a specific statistical framework in mind! Any suggestion is welcome! I am not too strong on Bayesian statistics though, so that would not be my first choice (although this could be a good time to start looking at that in more detail).
I can help you if you can explain the problem a bit better. Since you wish to use EM, there should be an observed (known) component and an unobserved (hidden) component. In your case, what is observed and what is latent/hidden? The mention of EM and Markov chains collectively bring to mind the Baum-Welch algorithm for Hidden Markov Models. If that is the case, there is a package called HMM in CRAN.
@Nameless I'm pretty sure that your interpretation of a 95%-confidence interval is wrong. In a frequentist framework, the value is *fixed* and the endpoints of the confidence interval are random variables. For a *single sample,* the confidence interval either includes the true value or not. To say a single confidence interval includes the true value with a probability is not applicable. See [here](http://stats.stackexchange.com/questions/11609/clarification-on-interpreting-confidence-intervals/11727#11727) and [here](http://stats.stackexchange.com/q/6652/21054).
What does "more dispersed than a normal distribution" mean?
The part about estimating % of variance uniquely explained is, I think, covered in the answer you have already.
@RobHall:  This is a specific instance of "The importance of hypothetical issues for imaginary data".  Cf. Wagenmakers, 2007, p.  784.  Wagenmakers specifically draws in the issue of transformations in the second column stating "in order to calculate a p value, you need to know what you would have done had the data turned out differently... this includes what you would have done if data had clearly been nonnormally distributed ... , p values can only be computed once the sampling plan is fully known and specified in advance".
Fitting process usually involves some kind of optimisation, so if you red-curve has some parametric expression, then yes you can fit it, using non-linear least squares for example. If the red curve is fixed then you can only calculate some metric which says how close your red curve is to the black curve.
@user30295 No, I am sorry. In that case you probably have to deal with multiple dependent variables and run one meta-analysis per dependent variable.
Nunnally has largely popularized those thresholds (emphasizing the distinction between individual vs. group-level decision, though); but there are so many flaws with its use in current research that it's probably not worth to worry about them :-)
For illustrations of pca, please look at the highest-voted questions with the [tag:pca] tag on this site.
@Peter That is far from the truth.  One can modify the geometry of the graph of the PDF almost arbitrarily without changing any specified (finite number of its) moments.
I'm gonna read it. It seems to me that when we have a limited integer->limited integer space ,the cut point value for a decision on integers(3 in e.g. X>3) comes from the algorithm behind that decision tree(such as ID3 algo. which uses Fayyad irani Multi interval discretization in Weka implementation) and this number (3) is not necessarily in  my feature values.In reverse if i depict a tree manually for representing my experience table(training dataset), i won't use this boundary split value,but i will use a multiple split at the node with equality comparison.
I think so (I am not familiar with the proofs)
Err, I meant if Mathematics would be more suitable, where I intended to post this initially.
I tried to download the forecast package but the Arima function cannot be found. Unfortunately, the arima function alone doesn't appear to be sufficient. Is there anything else you can suggest to me? R also can't find the forecast function. Also, how would I find the RMSE and MAPE values through this package?
Thanks, from what I'm reading this seems to answer my question. I'll look more into it.
Framing the task as the very general "multivariate investigation" won't be enough:  you'll need to narrow things down to a specific procedure and then you'll be able to find specific help on sample size / power analysis.
The equivalence holds only for certain models, e.g. random walk + noise ~ EWMA or local linear trend ~ holt-winters EWMA. State space models are a lot more general than custom smoothers. Also initialization has sounder theoretical bases. If you want to stick to random walk + noise, and you are not familiar with the Kalman filter, then you might be better off with EWMAs.
@Tom: 1) Can you say what 'bm' (in your edited question) is supposed to represent in terms of beta coefficients? 
      2) You computed Var('bm' - 'wf') assuming 'bm' and 'wf' being uncorrelated... which is probably false because your result differs from what we know to be true...
Why do the variables have different numbers of values?  Is it because some subjects did not answer the question?  To calculate any kind of correlation you need to know which of the six values in your "stationery" variable relate to which of the eight values in X.
@AdamO Tks. Actually, I was also looking for some insights on this. Could you explain, please? Is this approach wrong?
I think you may be confusing the apparent size of a "deviation," as *written in a formula,* with the actual deviation between two distributions.  In both cases the result is *remarkably* close to Normal. In case 1, the PDF is difficult to distinguish visually from a Normal PDF; all its odd moments are (obviously) $0$; its kurtosis is $73/25$, just a tiny bit less than that of a standard Normal ($75/25$), etc.  The fact that the Shapiro-Wilks test has *any* power to identify this mixture as non-Normal with a pseudo sample of 625 is remarkable.  The second case is similar.
One example is mentioned at the end of [this answer](http://stats.stackexchange.com/a/22350/919): Fisher's re-examination of Mendel's pea experiments.
@Peter, if the errors are not Normal the cures you suggest may not help, or may not be required. E.g. if the mean is specified correctly, and the errors have constant variance, Normality is a non-issue for inference on parameters in the mean. (See e.g. McCullagh and Nelder) Fitting a spline representation of a covariate helps the mean be "less wrong", but does nothing about non-constant variance. "Robust regression" (not use of robust standard errors) provides robustness to erroneous data, but can reduce the influence of the most informative good data points.
I'm not sure I follow. $\\mathbb{E}(S_N^2)$ is not just $ = X_1^2 + X_2^2 + ... + X_N^2$, and not even equal to that with terms like $2X_1X_2 + 2X_1X_3 + . . .$. I believe it is equal to the first equality in my question (I could very well be mistaken of course). [I do actually apply this answer to calculate $\\mathbb{E}(X_1^2)$ and so on in my $r=1, r>1$ cases above]