tttjjj/elt_bench huggingface dataset:
@ jbowman, you answered my question very clearly. Yes, I checked the return samples based on your formula:  For my cases, N = 5000, default burnin samples is 0.1*N = 500   (1) Case 1: the number of samples actually returned is 5000 - B/T = 5000-500/1 = 4500  (2) Case 2: the number of samples actually returned is 5000 - B/T = 5000-500/2 = 4750   (3) Case 3: the number of samples actually returned is 5000 - B/T = 5000-500/5 = 4900    (4) Case 4: the number of samples actually returned is 5000 - B/T = 5000-500/10 = 4950  Thank you very much! Best Regards,
Is it appropriate to focus solely on the 2008-2009 period (learning sample of the model) to develop an econometrics model earmarked for estimating what would happen in adverse economic scenarios? I gather I am simply repeating my question. As is, I think it is clear enough.
OK. Basically, the algorithm starts with membership assignments that are random, but there are close to G members in a cluster, and there are K clusters overall. We define the error function that measures the average distances between data in one cluster, averaged over all clusters. Going through all pairs of data systematically, we see if exchanging the membership of those two data results in a lower error. If it does, we update the lowest possible error, otherwise we undo the membership switch. We do this until there are no more moves left for one entire pass.
Cross correlation functions will allow you to do what you outline. Plotring te cros correlation of two time series will show peak at times of highest mutual correlation
I would suggest that this would be the more appropriate place for this question.
Your question may be related to [this](http://stats.stackexchange.com/q/55798/3277) one
this book doesn’t have a french edition?
@Corone I am assuming "Randomized Controlled Trial"
How it will come out is completely 'predictable' in the sense I think you intend it - no worries there.
+1, I especially like the subtle point that prediction intervals will be more sensitive to the distributional shape than confidence intervals for the mean.
Thanks for that and for the reference to Brewer.  I was aware of ColorBrewer but not that she was an academic.  Currently downloading some of her articles to read.
I added my Python code. I can't seem to get where my mistake is.
@Tal: Seems to me that you'd have to weight people based on how able they are to help you (directly and indirectly via their own friends). I don't think any of us can answer that. If all people are equally likely to help you (again, directly, or via recruiting their own friends), then a pure graph theoretical answer would be useful. Only you know that, however.
Your problem is not PCA problem but a wider _missing values trearment_ problem. If you're not familiar with it, please _read_ a bit on it. You have many opportunities: (1) delete cases listwise or (2) pairwise, or (3) replace missings by mean or median. Or (4) replace by random chosen of valid values (hot-deck approach). Or impute missings by (5) mutual regression (with or without noise addition) approach or by a better, (6) EM approach.
Have you read the introduction section of the [wikipedia page](http://en.wikipedia.org/wiki/Simple_random_sample) on simple random sample? There the difference between sampling _with replacement_ and sampling _without replacement_ is mentioned.
A first hint: $R$ and $\\theta$ are independent iff their joint density function is the product of their marginal density functions. A good place to start with this kind of problem is usually to try to find these functions.
To make progress, one would have to come up with an operational definition of "ceiling effect." Doing that in general could be problematic: after all, almost any measured quantity cannot realistically exceed some value, such as 100% in a test score or chemical concentration, the upper limit of what an instrument can read, and so on, so arguably almost all data are subject to some inherent upper bound. So, although the intended meaning of "ceiling effect" is intuitively clear from your nice examples, you can help us out by clarifying exactly *what* needs to be "removed" from your data, and why.
A "score" refers to an individual value of whatever outcome measure you are looking at in each visit, i.e., the dependent variable. So maybe for one model it is the gaze ratio you mentioned, or maybe a value representing amount of computer activity, or whatever. A "pre-score" is a value of this outcome variable during the "pre" phase, and likewise a "post-score" is a value in the "post" phase. Sorry if this confused you, it is pretty standard terminology in my field. What I mean about holding things constant is this:
Thanks. I have already looked into cmp. My question at this point is, whether I can extend biprobit  +ols for y (with selection adjustment coefficients) version as discussed in Madalla (p.278) or equation 19 of 
[this paper](http://www.bls.gov/osmr/pdf/ec050130.pdf) as triprobit plus ols for y (with selection adjustment coefficients). My model have two endogenous vars and one probit from two part(heckman), hence the need for trivariate probit. Also I need to model continuous var y with the selection adjustment coefficients as in biprobit case. Any reference to the papers would be appreciated.
Possibly a vector autoregressive model (VAR)?
@Roflcoptr Be aware that reporting only mean $\\pm$ SE or a 95% CI for Likert items, whose distribution are likely to be asymmetric (and this is your case), might be misleading. Another solution would be to report mean $\\pm$ SD *and* the frequency of observed response in the two upper responses modalities (4+5, or "agree" and "strongly agree").
In data that has a cross-sectional and time series components, it would be unusual to find one weight that fits everything. Statements (b) and (c) seem correct. Whether (a) is correct would depend on what phenomena the longitudinal weights were designed to weight for.
This question is not about obtaining specific datasets, hence I vote to reopen.
@rbatt: well, know the difference between the appearence of events in the different conditions. Looking at the graph one would say that one conditions has more events and that they start appearing earlier in time.
@AnthonyDamico, Thanks for the clarification on the issue of the correlational structure -- that is a helpful perspective. For some relationships, the clustering will most certainly be a problem. Assume that I ignored the issue of clustering in a ML approach but subsequently explored some of the correlational structures with and without the weight and stratification variables using more common regression techniques. Of course, direct comparisons couldn't be made, but it might offer some insight into the extent to which clustering affects the correlational structure. Any thoughts on that?
@user22374 Please follow [this procedure](http://stats.stackexchange.com/help/user-merge) to merge your accounts. Thanks!
Welcome to our site!  A high-voted answer is a great way to start out: I hope we will be seeing many more answers like this from you.
I have edited my question to provide this information.
By that reasoning another question which asks for the best blogs but lists other blogs in the starting question wouldn't be a duplicate of the question.
I think it makes sense to vote on all blogs to find out which are most popular including those already listed in the OP.
Is it fair to say that by "relevelling" you mean you are changing the base category for the factors?
...and of http://stats.stackexchange.com/questions/94337/what-problems-do-non-normality-in-predictor-variables-cause-for-a-multiple-regre/94363#94363
Can you be a little more specific in terms of data you have?
You might get more responses to this on Stack Overflow...
@MarkDollar May I kindly remind you that you can upvote response that you find useful? However, you'll need to **register** your account first.
I may be prepared to assume a distribution in various situations, and to check it for plausibility, but in real data analysis, I don't recall ever knowing the actual distribution.
How do you know that one is normally distributed?  What do you want to compare, exactly?
This is not a “multivariate” model as you only have one outcome/dependent variable. Fitting a model with several predictors is called “multiple” regression.
Extrema from cubic spline estimates can be obtained visually or using simple calculus of the fitted regression equation, standard errors of the post estimates can be obtained from the fitted regression coefficient matrix and its estimated standard error.
Could you provide more details on how you would want to apply evolutionary algorithms in this setting? What would your fitness be, what would your input/pool be?
By "PCR", do you mean *PCA* (i.e., [principle components analysis](https://en.wikipedia.org/wiki/Principal_component_analysis))?
That's right @whuber. I meant independent variables, and so- edited it.
In R, you use (assuming beta=1) `integrand<-function(x){(x^4)*(exp(-(x^2)/2))}` and then `integrate(integrand,lower=0,upper=Inf)` with results=
3.759942 with absolute error < 7.2e-06
"You have a number of independent data sets, which you are assuming follow the same family of distributions (in other words, a model) **but with different parameters**, and want some **automated procedure for selecting the appropriate family (or model)**." Good description of what I want. The question is: `a procedure for selecting the appropriate model` how?
A Chi square might be the way to go then. Your last paragraph is true. Fortunately I don't need to answer a question as such. For some reason I keep thinking about standardizing the response (number of degrees awarded) to both the total number of students per school and the time that each school has existed. Then running some sort of Glm, but I can't settle on an appropriate distribution for my data.  I'll have to give it some more thought, but the simple Chi Square, assuming a uniform distribution, might be the way to go. Thanks again.
@MichaelChernick: Could you please be a little bit more specific about how to adjust for multiplicity ? What kind of correction do you have in the back of your head ?
@ziggystar the `homework` tag now maps to `self-study`, with similar intent
It's possible to use them but not so directly as in, for example, random forest where absolutely no data preprocessing of the feature variables is needed. In fact they are totally inmune to monotonic transformations of continuous features. That being said SVM can perform very well but need a little knowledge on how to tune them, they have more complicated hyperparameters. Random forest is a very easy way to get acceptable results so I would go for this first.
Thank you! I suspected that the DW test showing was a simple coincidence. Because the distance between various regions was not specified and they were simply numbered as 1 to 15 without categorization.
Although, I'm still a little confused why you need error bounds or confidence on your z-scores. Usually, working with means is more interpretable. Can you elaborate on what your aim is and how you're calculating your z-scores (e.g., are you calculating them from the raw observations or from a mean)?
Just don't read the ggplot2 source. It will make your brain bleed
That python library seems perfect. Thanks.
I'll post them if I find any.  So far the docs have some examples of fitting networks, but no examples of prediction.  And some of the fitting examples have bugs.
A request for clarification: (1) The example matrix from the Chicago paper appears to show that M_{1,2}=19. However, I understood that by construction M_{t,j}=0 for all j > t.       (2) The reported OLS estimates include p_{2,2} = 0.823. But I had understood that p{2,2} = 1.00 by construction, since State 2 is an absorbing state.
One outside facility is measuring in a 6hr time frame, internal observations are over a months period of time. I agree with number two.  When you say bad-will among facilities, well there is even though the observation is done by professionals.  Our internal data for the two months that the external data is done does not match.  One reason that I find is the length of time is different and the rules are different.  The external observers go by a different set of rules than the internal observers.
i think you might be confusing {R-sq} and {adjusted R-sq} here.
Thanks Dan. If this is the case the answer is clear. What confused me is the textbook of Bisgaard and Kulahci "TIME SERIES ANALYSIS
AND FORECASTING BY EXAMPLE". They use a single linear regression model and they say that Y should be whitened using the AR coefficients of the AR(3) model of x (chapter 8, page 207). It was confusing especially with the presence of a second independent variable. Anyway thanks a million.
So true. I did say that the process needed to be geometric. Failing to transform can lead to errors in inference as well. I do not see where I was suggesting one ignore the assumptions regarding the conditions of valid inference.
About the invalidity of the test, the Wooldridge (Introductory Econometrics, 2nd edition, p. 253) says: _"If heteroskedasticity is present, this version of the [F] test is invalid. The heteroskedasticity-robust version has no simple form, but it can be computed using certain statistical packages."_ And the mystery continues!
Both PCA and CA are related and both can be based on SVD algorithm. The fundamental formal difference (not mentioned in @Gavin's otherwise deep answer) is that PCA decomposes relations between columns only (e.g. by decomposing their covariance matrix), treating rows as "cases"; while CA decomposes columns and rows simultaneously, treating them symmetrically, as cross-tabulation "categories". Hence the biplot left by CA and the quasi-biplot (loadings + scores) that could be plotted after PCA give conceptually quite different information.
See Section 9.3 and 9.4 of Cryer and Chan (2008). There is a free version of the pdf at http://carlossicoli.free.fr/C/Cryer_J.D.,_Chan_K.-S.-Time_Series_Analysis__With_Applications_in_R-Springer(2008).pdf (probably illegally).
Seems interesting to me that you can do classification this way without supervised learning.  It seems intuitive that there would be more accurate supervised approaches because the supervised approaches can take advantage of the added information in the labels.
Please edit this question to describe the data sets--the sense of "coerce" otherwise is too vague.  What is the purpose of your comparison in the first place?
Hi learner, welcome to stats.SE! I took the freedom to rephrase your question a bit. The edit is currently in review, but when it is approved, please check if it still matches your question. If not, feel free to revert the change!
The help discourages cross-posting. You should pick one best place for your post.
@raegtin, seasonal component that could be described by the weather conditions, temperature for instance.
Thanks for the derivation of likelihood function. It is clear that intercept term cancels out from the numerator and denominator.  Cheers
@onestop This prompts me to think whether IQR should be an interval or a scalar. From Wikipedia, it is defined as a scalar and is consistent with what I have learnt.
@Srikant Thanks.  Your formula seems to assume you have to use all k letters to form the word, but I don't think that's what the OP is asking.  (That's not how Scrabble is played, anyway.)  With that implicit assumption, you're on the right track but you need to modify the algorithm: you mustn't repeat the calculation for words in the dictionary that are permutations of each other.  For example, you mustn't subtract both t_{stop} and t_{post} in your formula.  (This is an easy modification to implement.)
indeed here there are no logitudinal data.
@whuber My comment has nothing to do with the purpose of data analysis and is probably not a fundamental disagreement either.  It really only has to do with the word justified. If you want to say that a good fit is valid justification for the purpose of data analysis I won't disagree.  If you you mean justified in the sense that goodness of fit implies that it is a valid description of an underlying process I would vehemently disagree. That is the sense that I took the OPs term "mathematically" justified.  So when I say that it cannot be justified statistically either
As far as I know, if you can't get any variation in other variables, you are out of luck. Sorry.
Assuming you have a Poisson point process the simplest possible would be the rate parameter $\\lambda$.
I think he wants 5 sec. binning. I also assume his X are integral and hence each X could get its own count over every 5 sec interval.
A single large outlier in the training set (and not in any of the CV sets) might cause the larger error. Did you check the data distribution?
Density is not probability. For a continuous variable, the probability of taking any specific value is effectively $f(x)dx$ not $f(x)$. (Consider a normal with mean 0 and standard deviation 0.1; what's $f(0)$ now?). This is addressed in numerous posts. In your situation above, a probability like $P(a<X<b)$ for $a<b$ and both finite will have some finite value strictly between 0 and 1. The answers [here](http://stats.stackexchange.com/questions/4220/a-probability-distribution-value-exceeding-1-is-ok) may be of some help.
Could you provide a reference to what you heard or elaborate on the context a little?
I think this one should be left open; true, it's about `R` but the content is inherently statistical
Here's another idea. Could you also get the country from the IP address of the survey takers? Most web surveys come bundled with that data.
$Y_1$ and $X_4$ will be correlated no matter what, as $X_4$ is a part of $U_1$, and $U_1$ is, in turn, a part of $Y_1$. So I don't see the importance of stating these correlations.
@Richard It helps to know that $\\sum_{i=k}^n \\binom{n}{i} (1-x)^{n-i-1} x^{i-1} (i - n x)$ = $k x^{k-1} \\binom{n}{k} (1-x)^{n-k}$.
@niandra82 I think the last error is because I forgot to use as( ,"STSDF") to my data in the krigeST() function. However, let me know if I answered your question...thanks for your help again!
Please make it more **clear and consistent** which are "classes" and which are "categories" (clusters). Prof/amature or the genres? Is it genres that you get from clustering and prof/amature is the labels that you know in advance?
This sounds like a problem for which meta-analysis may be well suited.
Also a note on confidence intervals on proportions -- most of the preferred methods for calculation will return an asymmetrical confidence interval (e.g. using the binomial distribution). See [the OpenEpi website](http://www.openepi.com) under the "proportions" option for some different calculation methods for CI for proportions, which you could cross-check with your SPSS results.
(i) why is 'algorithms' one of the tags? (ii) are the components independent?
(1) Not sure what do you mean by 'it gives me the same number multiple times'. If you have only 3 variables with only 8 possible values AND you are rounding to integers I would not be surprised if a lot of respondents had the same average. (2) Just as a suggestion you could also use the MEAN function instead of adding and dividing. See for instance http://kb.iu.edu/data/adjn.html
I think you don't understand it. If $F < F_{\\mathrm{crit}}$ then *automatically* $p> 0.05$. Correspondingly, if $F \\geq F_{\\mathrm{crit}}$ then *automatically* $p\\leq 0.05$. Alternatively, if  $p\\leq 0.05$ then $F \\geq F_{\\mathrm{crit}}$ and if $p> 0.05$ then $F < F_{\\mathrm{crit}}$. Further, under no circumstances can you say the two *populations* the samples were drawn from have equal variances. Whether the samples themselves have equal variances you can tell just by looking at the numbers - you don't need a test for that, but when they differ it doesn't tell you much of interest.
I was considering making it so that my sample had the same proportions as the total data set. For example I know a group of type A makes up 5% of the total data, I would select at random from my sample of A data so that it made up 5% of my sample
I think the key question here is, as you point out, how they performed the sampling. It's a fairly narrow target they are pursuing.
I am not sure how the weighting will affect the results, but if you're just looking for the raw counts, you can use the formula of RR and the formula of its standard error to derive two equations. Two equation, two unknown, and you would be able to find out what is the number of deaths in treatments A and b. Reference formulas [here](http://stats.stackexchange.com/questions/3112/calculation-of-relative-risk-confidence-interval). According to my calculation, the #s of deaths in group A in your first example is about 842; group B, 1433.
Okay.Thanks very much Randel.
This is a fair warning and always worth bearing in mind.  But it's a little harsh to insinuate (via "to correct") that the other answer was wrong: I believe the intention of @Charlie's circumlocution "trying to get at" was to indicate that observational studies do not "give" causation.  (It is clear from the body of Charlie's many excellent replies that he was not making such a mistake.)
Hmm - my question is not very well defined. The only thing I can do is pick *some* model for q() that permits setting parameters, and maximise the goodness of fit by fiddling with those parameters. That is - no matter what I do I will have to make some assumptions about what q() basically looks like.
@Ant's At least the 2007 videos are still accessible, and but for the videos, the 2011 course materials are also accessible.
@conjectures You need a negative sign in there too. I think we are all making similar comments in different style.
Data can be discrete without being restricted to integers. Or numbers, for that matter. It's always possible to *represent* discrete data with integers, but that doesn't mean the data can only take such values.
'Log linear models' is a term used to represent several different things. Can you give some clarifying context? Also, what do you mean by 'table representation'?
People trained outside of economics (say biomedicine or psychology) would have no idea what you are talking about ;-) Note that there are only two time periods (before/after), is all this really relevant or necessary in this case?
This is incorrect: the modes are all at the root of $6 x^4-27 x^3+33 x^2-17 x+3$ lying near $0.359187$ and are not equal to $1/3$.  This error is interesting in how it reveals one of the subtleties of the question.
Ah, the problem is precisely as reported by the error message: you have 6 levels of your within-Ss variable and only 6 levels of the wid variable split across 3 groups. If you leave out the effect of pH, then you can get your sphericity test, but this of course ignores pH. I frankly don't know enough about car::Anova's underlying mechanics to know why it is unable to compute the sphericity test in such cases when SPSS is able to compute it, but the developer of car is rather expert, so I wouldn't put it past SPSS to be doing something wrong.
I have also included `beanplot`.
CRM? (Yes, I know what it is. But, it helps to define such things within the question.)
You should give an example. It took me some minutes to figure out what you mean. Like for `1,3,4`  and `4,3,1` it's not 4+4, but 3+3 at $t=2$.
Ah ok, right, it just strikes me as... 'unsafe', to always use the same 2 columns of the U matrix forever?... I just wanted to make sure that was ok.
@EAguirre, re reading my answer i notice that what I wrote works for eigenvalues, not the diagonal elements of the matrix.
(+1) I like this answer. It took some time to me, but it seems I have been missing something in the question (*200 random samples* of size 25 each).
Given observational nature/retrospective nature, some form of propensity score is usually used. Annals of Internal Medicine has a couple of articles that could serve as templates.
Two examples per class is what I have. I'm not looking for statistical meaningfulness, I'm looking for a good fit.