tttjjj/elt_bench huggingface dataset:
I have 43 items, some measure the same constructs, therefore I wanted to use PCA to reduce the dimensionality, as it is also not so good to have so extensive dimensionality in cluster analysis. This is for academic purposes, therefore I need to somehow reason the validity/ stability of my results. When previously validated scales are used, I see researchers use CFA. I have deductively aggregated items from different scales. I am unsure whether its enough with just PCA and analysis of Cronbach's a or better use split-half test (either PCA on both divided samples or PCA on 1 and on another CFA)?
How long is this vector? One cheap and dirty approach would be to do multinomial sampling with this vector of probabilities enough times to form a data set, and then apply some standard approach. You could even do some 'smoothing' on the values by adding in normally distributed noise if you felt worried about underestimating variance.
For examples, see [pp. 20 and 38 here](http://faculty.washington.edu/fxia/courses/LING572/EM_collins97.pdf), [p. 85 here](http://www.amazon.com/Algorithm-Extensions-Series-Probability-Statistics/dp/0471201707) -- try "saddle point" in Amazon reader.
I think you understood it wrong. I have in total (40 countries * 5years * 4 quartes) observations. But still can you please elaborate on the Panel Regression? I only have column "tl" as the only data row. I mean what will I regress on?
Interesting question. Did (or do) you see anything relevant under the Optimisation CRAN Task View http://cran.r-project.org/web/views/Optimization.html
@Neo, you are welcome :). If you are satisfied/accept the answer you should accept it ;).
As far as I know, "pivot table" is a software device term and contingency table is a statistical term. Any electronic table where you can mouse-drag data categories from columns to rows, from rows to layers or other way, and do other flexible restructuring - is a pivot table. Contingency table is just a crosstabulation (most often, frequency) where cell entries tell about some sort of "affinity" between rows and columns. It can be on paper, not only electronic.
tnx for the help.
@Cam.Davidson.Pilon: I agree that the integral should be well defined. " it is integrable if either g(x)≠0 in [0,1], or x is dominated by g(x)." I wonder why x is dominated by g(x) will make the integral well defined? You can have $\\frac{0}{0}$ in the integrand. See the comments after Jonathan's reply.
I really appreciate your help. I have tried the method of moment. Please see the updates in my OP. I cannot find the functions in the R packages that you mentioned in your answer to my other question can evaluate the mean and variance for a given truncation point for gamma distributions. Any help would be appreciated !
As so often in your posts, you seem to be assuming that a given term is always used in the same way by different authors in statistical science. "Model checking" is not rigidly (or rigorously) defined anywhere so far as I know, and if it is, other people don't care a hoot: they just use it as strictly or as broadly as the spirit takes them. It's not like "isosceles triangle" or "Abelian group" where there's a precise definition.  As for the last question, are you really unsure whether $R^2$ is used in regression? If so, your question should be downvoted for lack of research effort.
By term "covariates", SPSS dialog boxes mean continuous predictors (whereas categorical ones are labeled "factors"). So, SPSS uses the word in wider sense than it is used in ANCOVA (Analysis of covariance).
I understand your concern, but I think it would be logical to have a higher R2 after weighting  when there is really a relation between your effect sizes (ES) and size.
Usually there is little difficulty with interpretation when a *predictor* is transformed--certainly not in comparison to [interpreting a transformed predictand](http://stats.stackexchange.com/search?q=power+transformation+interpret). If you read these threads related to interpreting transformations, you might learn all you need for understanding your own situation.
@EdM, z is missing for all individuals, and Yes--the "other data" contains the all the values of the variables in the main data.
@user4119 I've tried to reword a little your post, and I corrected your $H_0$. It will be helpful if you can confirm that (a) parental response is the outcome, scored as a binary response (positive or negative "case"), (b) the scale you used is already a validated and unidimensional scale, that is working with a totalled score makes sense and reflect a single underlying construct (and all items were correctly reverse-scored, when needed). You said you cannot test for a gender effect because N's are even: it doesn't make sense at all.
+1, it is basically the same as my comment, but much nicer written.
That is correct: instead of "nugget" in your formula you should write nugget$(h)$ to indicate that the value is $0$ when $h=0$ and otherwise equals $0.1343$. Because the Gaussian model is $0$ when $h=0$, that makes all diagonal elements $0$.
What about `boxplot(your.five.data.points)`?
I'm not sure if I fully understand your situation, & I don't have SPSS, but in general, MANOVA can be thought of as an ANOVA performed on a specific linear combination of your DVs which is basically never equivalent to exactly one of your DVs w/ the others dropped.
Guys remember my dataset is highly unbalanced. I have very few of them are 1s not even 10 and like 19990 0s
It can be hard to hear that final "D", so that's one issue. Also, it isn't clear what exactly is being squared. One doesn't talk about "chi" itself (unlike R squared, where it is clear that it is R, squared.
`Kruskal-Wallis... assumes equal variance` This is incorrect statement. If you've picked this from Wikipedia please know that the article is inaccurate. See Discussion sheet there.
Hi lurie, I don't think that as it stands "Was the correct analysis used in this paper?" really counts as a suitable question for CV; if not for the question in your title, it would be likely to be closed as unlikely to be of interest to other visitors. If you could instead describe the circumstances you're interested in and ask what analysis is suitable (with perhaps a link to the paper still, as background to your question), that would be a better fit to CV, I think.
Good advice, except that observations in a series of years are most unlikely to be independent, which is an assumption here. That assumption applies to the Wilcoxon test too. Box, Hunter, Hunter "Statistics for experimenters" is one of the few introductory books to explain this well.  It is hardly to recommend any test without knowing about the underlying process.
Are zones in some order? If so, which of the left side zones are closest to the right side (and vice-versa? If not, which zones are adjacent to which?
What is "significant" component or variable for you?
You can always construct a (standard) Markov model out of the vectors $X(t) = (S(t-n+1), S(t-n+2), \\ldots, S(t))$: now each state $X(t)$ depends only on the values found in $X(t-1)$ and you're off and running.
Unless you've eliminated the intercept, the log odds for the reference category (level 4, in your case) is equal to the intercept, not 0. If you'd prefer to use a different reference category, I'd suggest searching SAS documentation for how to change the reference category, although this will not (mathematically) change the model or the fitted probabilities in any way.
(+1) Your solution is at least more elegant than mine because you don't duplicate the grouping variable!
To me, this is by *definition*: correlation is one of those properties of pairs of datasets that does not change when either (or both) are shifted or rescaled.  Among all such properties that can be expressed in terms of moments it is the simplest. (That defines it up to a nonzero multiple which can be pinned down by requiring that "perfect" correlation (of a set of data $(x_i,x_i)$) equals $1$.)  This shows that your question is incomplete: you need to give us *your* definition of correlation.
Actually my reply got posted by mistake before completing my comment. What Im jst confused abt is that usually without performing a diff-in-diff analysis, there are variables that determine the dependent variables. I was wonderin if I shld include them along side treat, post and interaction.
Post some data as that might help me "seeing" the problem that you have. I am quite familiar with the histogram highlighting the outlier and I might be able to provide you with some guidance about an efficient way to detect these anomalous cases using statistical methods in lieu of inspecting histograms by a human evaluator. See a recent discussion on detecting randomness http://stats.stackexchange.com/questions/12955/how-to-find-out-if-a-set-of-daily-measurements-are-random-or-not/12957#1  You of course are trying to detect deterministic violations of randomness.
@gung - Unfortunately, "inattentiveness" implicitly carries the "-" sign with it.  So an improvement in inattentiveness is less inattentiveness, or a negative change in the metric.
You do not seem to be addressing the question concerning $R^2$ of the regression of predicted against observed values.
thank you for the suggestions!
Nice answer (+1). One detail: Perhaps you were trying to draw a connection to the Benjamini Hochberg procedure and its assumptions, but the section on Bonferroni correction is implicitly assuming independent tests, which is unnecessary and, in a sense, misleading. I would argue that showing the general case is actually *more* enlightening simply because it readily accords with common-sense notions and also shows, in a certain sense, why you need stronger assumptions to get a procedure with strictly better performance.
@MudPhud: Did you get any errors? Try typing `lrt.sim` to make sure they're not all zeros, in which case the most likely culprit would be that you don't have the package `lme4` installed.
What have you tried so far?  Presumably this problem has come as part of a course or a text book, so are there parts of the instructional material that look relevant but you can't follow?
yes and also z test. For correctly adjusting the alpha (something which should be Always done but is ignored vastly) you should calculate the power of your statistical test. Google "power calculation for kruskall-wallis test" (or similar terms). A part of the formula for this purpose is the alpha itself. So knowing the other parameters (sample size etc.) you can compute proper alpha.
Because of the comment I made. Even with such an oracle, how do we exactly trivialize the model selection process. That is the aspect that is still no clear to me or the part that I am not convinced about. It seems to me that, even if we had such an oracle, I still can't really see a way of trivializing model selection without having a turning machine that might go in an infinite loop.
Quite welcome! In that example, you would choose one category to subsume into the intercept, say `latino`, and dummies for the other two. A 1 value for the `caucasian` dummy indicates a Caucasian respondent, similar for the `black` dummy variable. A 0 value for both indicates a Latino respondent. Make sense?
@Corone, yes, it is for simulation purposes. I think I have changed my plan a bit, but still have few roadblocks to deal with. I will send a new question, about this. Thanks.
"(...) 16 rows and 6 columns": Are you saying you are using an FA model with 16 subjects and 6 items?
Almost. You can't have a dummy for every day of the week plus an intercept term because that generates multicolinearity. The easiest thing is to just include 6 day of the week dummies, with the intercept acting like the dummy for the seventh day. This formulation is (generally) much preferred to your first suggestion.
If you _actually_ have a sample where its min value is 1 and its max value is 10, then simply, this sample cannot be a realization from the assumed distribution, and you have a case of misspecification.
$2p-1$ is only positive for $p\\geq 1/2$, so the result $2p-1$ does not seem correct for a probability.
To be honest, I don't catch what you mean by "expressive" and "constraints". All difference between them comes from their formulas. Try to reflect on it or test some data, to see.
Thank you so much. That matches almost perfectly.
@picakhu, the best way to do that in Oo would be to use the arithmetic functions to solve the equation (and perhaps you could ask another question if you have trouble with that)
Depending on how this suggestion is interpreted, it may amount to either Fisher's Exact Test or Barnard's Test.  See http://en.wikipedia.org/wiki/Fisher%27s_exact_test .
Agreed; if you are worried about the efficiency of your coefficient estimates, using "clustered standard errors" might be fruitful.  But as mentioned, this is a separate (and minor!) issue compared with your regression discontinuity experimental design.
Neither "limited" nor "overdispersion" imply "bounded."  The Wikipedia article on [overdispersion](http://en.wikipedia.org/wiki/Overdispersion) points out that the negative binomial is a gamma mixture of Poissons, allowing it to handle overdispersion.  The article you cite is in a nascent phase; it uses "limited" only in a vague colloquial way that one hopes will be improved as the article evolves.
Yeah, this is the first thing I thought of.
My head hurts.  Can you write this with shorter but more sentences? (sorry).
Another thing Gelman suggests is trying to estimate a model like P(y=1) = 0.01 + .98 logit^-1(XB), which bounds the minimum and maximum guessed probabilities. This will mechanically put more weight in the tails but of course you can also make the .01 and .98 parameters to estimate.
Welcome to the site, @Emily! Can you reproduce these results in any other software? CFI relies on a baseline model, and with observed exogeneous variables in your model, the default model with no correlations may not be a very sensible one. But RMSEA of 15 is just weird.
If you want to capture trend then single time trend variable is the only approach, I'm not sure you can use dummies. If you want to use seasonality then you could use dummies. for instance if you have 12 months of data then you could use 11 (12 -1) dummies to capture seasonality in addition to continuous trend variable.
Can I just say, this is a fantastic problem.
I have not dealt with effect size before, but it appears to be a good idea.  I will examine that.  Thanks!
Interesting. I haven't checked R.. E.g. SPSS doesn't provide CIs for intercept. Please see updated question
A more fundamental concern is that PCA on the independent variables gives no information at all about the dependent variable. It is easy to construct examples where this PCA approach will eliminate every variable that matters and keep only those that don't! To see what's going on, let $(X_1,X_2)$ have a bivariate Normal distribution with variances of $1$ and correlation $\\rho\\gt 0$ and set $Y=X_1-X_2$. A PCA on data from $(X_1,X_2)$ will identify $X_1+X_2$ as the principal component and eliminate $X_1-X_2$. Regressing $Y$ on $X_1+X_2$ will be nonsignificant.
Thanks.  It is clearer.  Be aware, though, that many radically different answers are conceivable, any of them potentially correct.  Selecting a good approach depends on understanding (or analyzing) the relationships among the measurements and the object state, which may be nonlinear, have heteroscedastic and correlated errors, etc.  Therefore, the more information of this nature you can provide, the better you can hope the replies will be.
Welcome to this site! Could you summarize the main points of the reference you provided? In case of link rot, that might be useful for future readers of this thread.
Could you please show what your code looks like?  It will help us understand why you are getting the error.
See my answer to your cross-post: http://stackoverflow.com/questions/18972389/how-to-get-the-centroid-of-one-cluster/18976153#18976153
And in case where I have an all-0-vector in my dataset, what should I return ? (a vector where all values are 0)
What seasonal time series model did you use, and what did the residual series look like?
If I understand this procedure correctly, it takes default choices of kernel type and width as correct. To my mind, the uncertainty associated with kernel density estimation is not just a property of the data but also very sensitive to whether I am making good choices of procedure. Delegating these choices to a program is also a choice.
As I use the same classifier on two other similar data sets (same size) and those have an AUC of around 0.5, is your option 4 most likely?  The main difference between those data sets and this one is that the values here are in general much smaller.  Could this offer an explanation why the AUC (and thus SVM (linear) classifications) are much worse for this data set?

Though my test set of 20 data points would make your option 1 feasible too.

(validation set to combat overfitting was used, so I don't think option 2 is it.  Nor option 3 as it worked for other data sets).
Typical solving approaches for nonlinear SVM are not easy to distribute.
@Steve it's still not remotely clear why you are suggesting multinomial regression and your link to wikipedia doesn't help to explain your reasoning. Nowhere in the problem set-up was it suggested that the response variable is categorical. Nor does multinomial regression have any inherent variable selection properties. Your answer to number 1 ("you should ask and have answered why the data you were given is not in harmony with your coworkers beliefs") isn't really a satisfactory response either, since this was essentially the question *you were prompted with*.
It depends on the consequences of the amount of non-normality for the technique that you wish to use.
I would not interpret the pattern of residuals in the first model as heteroscedasticity, but instead as a possible non-linear trend.  Applying a robust smoother (such as LOESS) to the residuals will help you see that. If the smoother is close to a straight horizontal line you can suspect heteroscedasticity (and test it by smoothing the roots of the absolute values of the residuals), but if it wiggles around appreciably, you have detected a *systematic variation in the average residual.*
@bobfet1: I'm assuming the asker has 2 distributions of money, x and y.  From `?wilcox.test` you can see how to run this test.  For example, if the null hypothesis is that the 2 distributions are the same, use `wilcox.test(x, y, mu=0)`.  If the p values is less than 0.05, you can probably reject the null hypothesis and conclude that the means of the 2 distributions are different.
Thank you. Could you expand a little more about what you mean by "$\\chi^2$ performance of the likelihood ratio test statistic" ?

In this module, the regularity conditions are discussed but not in great detail. I believe the information given in the question (the pdf and the statement about the data collected) imply that the conditions are satisfied. Perhaps I should have just said (bearing in mind that this is all for just 1 mark) "from the pdf definition and data the regularity conditions for asymptotic normality are satisfied" ?
I will run a Monte Carlo - Simulation tomorrow to verify that (boy, I am tired ;)).
Welcome to the site, @Susie. *Please don't cross-post*. That is against SE policy. Decide where you think this Q is more appropriate (probably here) & delete the other.
I added an in depth answer that lays out the solution I came up with. I'm not sure if my solution is correct, but it should give you an idea of what I'm looking to accomplish.
Thanks a lot Paul for the inputs.Will definitely work on it and get back to you.
many many thanks for your answer you said Aggregating highly positively correlated series doesn't do as much for you as aggregating less positively correlated series, but why? why aggregation works like this? could you please explain what do you mean by saying all other things?
Actually i use regression on Libsvm, I editted again as regression. I also used this aprroach not for 14 days but for 3 days like:
X (day 1 ).......	X(day 2).........	X (day 3)........	Y(day 4):::::
X (day 2)........	X(day 3).........	X (day 4)........	Y(day 5):::::
X (day 3)........	X(day 4).........	X (day 5)........	Y(day 6):::::

…		…		…
X (day 48).......	X(day 49)........	X(day 50)........	Y(day 51)

But the output Y (day 4,5…51) fitted the last X vector (day 3,4…50) this time. Is that the same approach with yours?
For another duplicate also see http://stats.stackexchange.com/questions/13850.
@smillig That's an interesting reference. Gradient descent seems like a poor strategy for solving the ML equations, though: it's just too easy for it to lock into a local maximum that is not a global one.
I find significant autocorrelation in the residuals from calculating the rolling return in such a method
Alternatively, `theme_get()` provides you with the same reference in the console.
This question is answered (in passing) at http://stats.stackexchange.com/a/49174/919.
Last time you asked about this, I suggested comparing ECDFs. Your use of the barplot has a number of issues, but a similar plot could be constructed that avoids most of them.
This seems like a typical homework/self-study question, and as such should have the `self-study` tag. See http://stats.stackexchange.com/tags/self-study/info
thanks a lot for offering to help. We have a couple of students in our lab who may look into this.  We'll let you know as soon as we figure things out :)
@user333 Could you work on improving your question, please? As it stands, it is difficult to determine exactly what question is being asked, and what would be the purpose of such a function *in a statistical context* (ok I guess it might be related to your previous questions, e.g. on random walk).
In SPSS, percentules are in Frequencies procedure. And yes, they obey frequency weighting variable which weights the dataset.
@Floris (1) The population variance is known, not estimated, so a t-test would be unsuitable even if this was a testing situation. Those are for when the standard deviation is estimated. $\\quad$ (2) In any case, the question is *not* answered by hypothesis tests of any kind. $\\quad$ Your first comment is not helpful.
Because this is such a specific application, specific information beyond "a lot of variables" is needed. Although you don't have to explain them all, it is important to know what kinds of variables they are, how they are measured, what aspects of those measurements you have recorded, and how much data you have.  Your edits to the question to clarify these points would be most welcome.
I do want to compare partitions of different k. What could I use? What's a good idea? thank you for helping me so much. @ttnphns.
That applies to modelling in general. Interpreting the model in your own special way is not restricted to economists, as far as my experience goes.
If you had all the data in the incomplete system, presumably you wouldn't expect it to be exactly identical to the data from the complete one, simply due to random variation. Is that right? If that's the case you should not use the complete one as the expected distribution. If the missing data is at random, you can do a chi-squared test that they both come from the same distribution. (There's still not enough detail to be entirely clear about what you actually do have. I've had to make guesses.)
The brilliant part about Sage is that it is essentially _the union_ of a number of great free tools for mathematics, statistics, data analysis, _etc._ It is more than just Python; it has access to R, maxima, GLPK, GSL, and more.
@JeremyMiles: Totally agree on that. Especially as a PhD student, long periods (5-6 months+) of review waiting time are disastrous. I am totally against rapid-fire submissions (and actually that's the editor's work to stop) but when on the other hand when you end up with something like a 10-month+ period for *submission-corrections-review_of_changes-acceptance* you really get demotivated.