tttjjj/elt_bench huggingface dataset:
Hi.Thank you for your comments. I've visited your website which was very helpful. Now I figured out how to conduct item parceling and path analysis using SPSS and Amos. My another question is that whether the factor that was not a unidimensional in EFA result can be used to create parcel. Several articles have mentioned about this issue and I wonder whether the factor must be unidimensional..
The criteria (such as Kettell's scree plot, Kaiser's rule etc.) are based on _PCA's_ eigenvalues, not the eigenvalues or extraction sum-of-squared loadings that you get at factor extraction. So, you do PCA first and decide on the number of factors. Then you proceed to FA. Tre problem of the number of factors has nothing to do with choosing this or that extraction method.
It's a little nitpicky but the CLT doesn't tell us that the 'sample' itself will be normally distributed.
Ok, so I could really just use the mean() function on my original data for mu...  that's what I was thinking but for some reason it seemed too simple.  I think I skipped over this line: nb0 <- glm.nb(Days ~ 1, data = quine)   of your explanation previously because I thought what you had might not apply to a data set with multiple predictors.  I now see what you were intending.  Thanks for all of your help!
You might as well mark your own answer as accepted.
It's unclear what kind of answer you're after. Do you know what likelihood is, for example? If not, better to find out that first.
Sorry, of I have no good suggestion off the top of my head. The problem is that the data is not i.i.d. anymore. One simply way of getting rid of the autocorrelations it to train an autoregressive model and subtract the prediction from the datapoints. This is essentially whitening.
As I keep digging around there seem to be essentially two camps... those who have a 'better approach' via a pooled error term and those who want to adjust the p-value according to some sort of multiple comparison procedure (Bonferonni, Holm... etc etc... there seems to be more of them than I have fingers).
No, of course not: some of the singular values in PCA can be true zeros, for instance. This is less related to the "sparseness" of the data than it is to whether they "fill out" the dimensions used to record them.
@Ran, it is difficult to determine what the problem is without having additional information about the size of the training set, nature of the attributes etc.  However one thing comes to mind which is that NB has a problem if the conditional probabilities are ever zero, as if that happens, the output will be zero regardless of the values of any other attributes.  Are the probabilities for the minority class always exactly zero?  If that is the case, that may be the problem, in which case using the Laplace correction to estimate the conditional probabilities might help.
What is your current knowledge about it from student point of view?
@2 is clever.  It might even be better to draw correlated random subsamples: e.g., randomly partition the data into $k$ parts of approximately $n/k$ values each and run $k$ PCAs for a computational cost of only $1 / k^2$ of a full PCA.
I would also use seasonality for 1 year cycles (linked to the seasons) but wanted to check with you guys. Do you think that monthly patterns (for instance end of month effects) can be called seasonal patterns?
Thanks Placidia. When I come back to this at the beginning of next week I will be pressed for time as well as stretching my brain cells back to my graduate level econometrics, so I want to keep this as simple (yet as accurate) as possible. I think after poking at the data I will either count the less trusted data the same as the good data or just not use it. In fact, as I sit here right now it is not obvious what sign would be on the coefficient for marital status. Maybe I will get lucky and it will be not significant.
I once asked the same question on the R mailing list, and didn't get a response.  I'd suggest you to change your title since your question is regarding "post hoc analysis of chi square - to detect the cause of the significance" (a shorter titles then the one I proposed would be better :) )
Hello, thanks for your reply. I hope I got what you are saying. I do not want to build the tree manually. I need algorithm to consider a particular variable only once. So if var A has been used to split the records at the first level then var A should not be used in any of the subsequent sub-trees. Essentially I am trying to show graphically, which variables are more significant in splitting the records.
@Joe: Good point - bias is defined in terms of the expected value of the sample estimate. All the same the asymptotic convergence at issue here is to a normal distribution whose mean & median coincide at the true value of the parameter.
I'd recommend the r-sig-phylo mailing list (https://stat.ethz.ch/mailman/listinfo/r-sig-phylo). Even if you are not using R for your analysis, you will get very good answers to your question.
I am looking on events which are sequence pf at least 4 events. where 3 of the first are identical.
@Raskolnikov and Elvis (+1), note also that standard continued fractions for $-\\log(1-z)$ truncated at the second convergent yield $-\\log(1-z) \\approx z/(1-z/2)$ and this approximation is always better than the second-order Taylor series expansion for the range of $z$ of interest.
Thanks, but the title is explicit. I am not looking for add-ins.
@JoelW. If you start out with a random sample and manipulate it like that what you are left with is no longer random.  So there could be bias in making inference that you can't quantify.
Monte Carlo in 1908 - men were men in those days...
What has U got to do with this problem?  And is this a homework problem (it is useful to know so that the answer can ensure it focuses on teaching you how to solve the problem)
It might be worth noticing that the relatively large values in the first dimension dominate the cosine similarity scores. This implies that the points most dissimilar to both `point1` (first component is 100) and `point7` (first component is 20) necessarily will have first components as close as possible to (100+20)/2 = 60. The best candidates are `point2` and `point5` but definitely not `point4` (which coincides with `point7`!). This naturally raises the prior question (which ought to be settled before we go on), *what is your similarity measure trying to represent?*
Demand sizes is the sizes of demand when demand occurs, Demand per period is all periods including the demand occurring and non-demand occurring. The time bucket is 1 month, the history length is 22 months, and the lead time is 8 weeks
Are you familiar with mediating http://en.wikipedia.org/wiki/Mediation_(statistics) or moderating http://en.wikipedia.org/wiki/Moderation_(statistics) variables? Darn I cannot get those end brackets to form part of the link.
If there is an underlying  Poisson process or a Negative Binomial Lévy process, the distribution of the number of events within a period can be related to the 
effective duration which may be known, possibly with error. For instance, if you  know that within a month the process was only observed during 20 days, this can
be used in a GLM with appropriate link.
Has it finished? It is supposed to produce a bunch of graphs, and it might be slow indeed on a large data set.
I think this post reveals that you are a bit too focused on the p-values involved in your interaction.  I recommend focusing more on the values of your focal regression coefficient (and the coefficient's interpretation) at the levels you choose (e.g., plus or minus 1 SD) of the other.  There are many reasons why focusing too narrowly on p-values can be misleading, especially when it comes to the interpretations of simple effects.
@Charlie Thanks for the response. I am VERY new to quantitative analysis, it Stata was just covered in my first statistics course. Can you explain a little more what you mean?
@AndyW This is more of a general question, not specific to any field. I haven't seen this frequently either, which was my motivation for the question. Something similar is described in this 1998 paper http://www.leg.ufpr.br/wiki/lib/exe/fetch.php/projetos:mortalidade:bibliografia:1467-9876.pdf in equation 2.4 but other than that, this doesn't seem common and I'm wondering why.
is it worth editing your comments above into your question? the context is pretty helpful
-1 This sounds more like a self advertisement than a helpful reply.
I understand, thanks. This is a slightly different question in my eyes in that it's asking about the validity of the application of a statistical approach as opposed to the interpretation of a specific model, but you are right that they both come from the same problem that I am having. Maarten happened to answer both questions, so I will clarify with him and anyone else in the comments to the other question.(http://stats.stackexchange.com/questions/57031/interpreting-interaction-terms-in-logit-regression-with-categorical-variables/)
@BobC, is the rough model I gave for the timing asynchronicity close to what you have? I'm thinking it's "random initial offset" + "linear error" where the latter is due to a small constant difference in the sampling interval between your two devices. Then there is some additional small random error, say due to interrupt processing of two different uC's on top of that.
Ah, one thing I forgot: the Poisson jump size is also multiplied by a random variable X that equals 1 with probability 0.5 and -1 with probability 0.5.
@DWin, yes, the rendering is a bit funky. I thought you might have intended the comment for another post since this one has no votes (up or down) and so the comment seemed to be out of place. Cheers.
Definitions are over-rated, but I don't think you can make real progress here without some sharpening of terms. Notation doesn't help if it's not explained. If correlation is more general than linear relationship, then you seem to be saying that relationships between variables must be explicable somehow. Fine, but that's too weak a statement to be useful. There are shelves of books and papers on this. Judea Pearl. 2009. _Causality_ Cambridge University Press is more aware of discussions across several disciplines than works that are discipline-bound (e.g. in philosophy, physics, economics).
...just in terms of how people would think about their answer choices, their choices between verbal and numeric rating systems probably being made on a preconscious / automatic / less-than-deliberate level most often across participants with varying degrees of self-awareness, attention to detail, or "[psychological mindedness](https://en.wikipedia.org/wiki/Psychological_mindedness)". Some amount of learning occurs over the course of filling out a long questionnaire. IIRC, this leads to more consistent (maybe biased) answers as items begin to blur together. </ wild speculation and conjecture>
Karim, are you really sure you would use an estimate of the population standard deviation in place of the *standard error of the mean* in the formula for a confidence interval? As far as your last remark goes, it raises a subtle but important issue: the probabilities of observing such weights are not assessed with confidence intervals but with *prediction intervals.* (If you haven't heard of the latter, search our site: there are explanations and formulas.) Given the elementary nature of these questions, I doubt that's the intended interpretation.
It's unclear what you are doing. How do you obtain $2^{31}$ permutations of $31$ things (when there should be $31!\\gg 2^{31}$)? What precisely do you mean by a "permutation," then? And of what use is permuting the $30$ control images if they simply remain control images?  And how is it possible that a *single* permutation may have a "low enough p-value" when in fact the p-value needs to be computed *after* the entire permutation test is conducted?
Good clarification! Thanks for your input.
Comments about how to improve this reply would be appreciated.
It looks like a Sankey diagram - [stackoverflow][1] has a topic on them. \\n\\n\\n  [1]: http://stackoverflow.com/questions/9968433/sankey-diagrams-in-r
Are you familiar with the Bradley-Terry model? It's probably the "canonical" approach to this problem, but it uses only an inherent "team strength" parameter. Generalizing ever so slightly the basic idea, if you don't have an indication of home field, then you can always use the *difference* of the predictors to enforce the symmetry you desire. For example, $\\mathrm{logit}(p_{ij}) = \\beta^T (\\mathbf{x}_i - \\mathbf{x}_j)$, where $p_{ij}$ is the probability team $i$ beats team $j$. Of course, you can't have an intercept in such a model. (see continuation)
I already tried the 2sls approach, but the results were not convincing (not in line with theory). Thanks for the link I will have a look
Spelling out which R function you used is always good practice.
Welcome to the site! As this site is read by many for whom English is a second (or third or fourth) language, it's best to avoid slang like 'plz' which I changed to 'please'. Also, it might be good to spell out RBF and SVM. Abbreviations can suffer in translation.
I think, given the data of a fixed No of points in a fixed No of dimensions, the greater is K of random initial centres the smaller is the room for a considerably suboptimal solution to occur. After all, if K = N of objects, the "solution" is always the best :-)
+1 for posting a useful set of hints that gracefully addresses our objective of providing helpful replies to stimulate thought.
(+1) [Linear Models with R](http://www.maths.bath.ac.uk/~jjf23/LMR/), by Julian J. Faraway is also a good starting point.
@Glen_b: And isn't the first version less appropriate when the mean is not fixed across various sets of ratings that need comparing? Or have I misunderstood your answer?
I'm doing some tests in document clustering in weka with Euclidean distance as the distance function. But my problem is even if I give a mix of unrelated (topically) and related instances during tests (even some empty instances), they all get clustered into the same group. What could be the reason for this?
@Nikhil There is no guarantee that the forecast will lie between 1 and 6 unless some form of a deterministic trend was found which is to me very unlikely given the nature of your stationary data. Secondly I am unaware of a SPECIALLY TAILORED model for this kind of data. Note also that the ISI has been a customer of ours in the past but I don't believe that they were concerned with your very interesting and well articulated problem statement. You might ask an ISI faculty member who is familiar with time series analysis if they can help you and perhaps share that information with the list.
I've included my code.
I've voted as a duplicate, and the question about analyzing pre-post data comes up quite frequently (see the related items on the right hand side of the screen). If the prior linked question does not suffice, please edit this question to make the differences clear.
Yes I would say that if you have 102 days and 86 companies , 187 anomalies might be a reasonable count. Just because this might be a small percentage DOES NOT MEAN that one shouldn't adjust.
@amoeba Yes, updated with a proof of that.
@Glen_b As I described above, the patients whose brains are believed to be "abnormal" are compared to the mean of the healthy people, so that is what that mean is used for. The main reason for this is that DTI (or MRI in general) can't be calibrated and the (absolute) values that you receive depend heavily on the scanner that you use.
I think "stable mean" is therefore pretty much equivalent to "close to the 'true value'", i.e. the value we would get if we could use the entire population of the earth as a control group.
Did that help?
I am not sure though that the solution offered would work. For we can write $E[obs] = (1-w) \\cdot E[obs|d=1] + w \\cdot E[obs|d=0]$ for the value of the unconditional mean, and my guess is that it depends on $w$ whether both are substantively equivalent. I don't know however.
$k$ independent trials per unit? What is a unit in your example?
When is it better to use accuracy instead of ROC? also when we use accuracy metric, does train function provide percentage cutoff which is giving the best accuracy or does the function only provides accuracy? isn't information provided by ROC always contains information given by accuracy and in addition to that quite a lot more information? for example train function will provide output that for a specific values of tuning parameters accuracy is the best. But in the same case ROC will provide what is accuracy over various cutoffs for the same value of tuning parameters
+1.  Didn't see all this earlier - Thanks for noting the issue as well as the crash/mini course on stata notation.  I took your first comment to imply the interpretation was mistaken and answered in a very general sense.  I'm glad you were more persistent and that kwak figured it out.
Wow, autocorrelation it is :-) The results of -1 to 1 is just perfect. I just installed R and will play with my real data over the weekend and look if the results is what I expected. Thanks Aniko! (Oops, I just saw I need a reputation of 15 to vote up)
I'm well aware that the problem is not properly solvable without p2. But my questions were specifically aimed at which regression methods that are able to detect and handle different "distributions" within the data. Any comments on this?
Unless you have an outcome variable you want to model, it sounds like a multi-attribute valuation problem: see answers [here](http://stats.stackexchange.com/questions/9358/creating-an-index-of-quality-from-multiple-variables-to-enable-rank-ordering/9361#9361), [here](http://stats.stackexchange.com/questions/9137/quality-price-trade-off/9139#9139) & [here](http://stats.stackexchange.com/questions/3201/how-do-i-order-or-rank-a-set-of-experts/3218#3218).
Regarding your first question, this problem is sometimes called *weekend effect*. In my opinion, the answer is context-dependent. For instance, this question makes a lot of sense in the case of stock returns. See for instance [here](https://umdrive.memphis.edu/cjiang/www/teaching/fir8-7710/paper/Stock_Returns_and_weekend_effect.pdf), [here](http://www.jstor.org/stable/2327716), [here](http://www.jstor.org/stable/2328067) and [here](http://www.jstor.org/stable/2327945).  But I am not sure if this effect applies to other contexts.
I just saw that I lost a sentence between my first and my second comment, namely: "Under some conditions that you will find in the textbooks on linear regression, regression has very useful properties."
If you only wanted to know how to do this in R, this question would be off-topic for CV (see our [help center](http://stats.stackexchange.com/help/on-topic)). However, I suspect there is a good deal of real statistical information that you may need. Eg, comparing AUCs is not generally the best approach & you wouldn't use a t-test to do it. Can you say more about your situation, your data & your goals?
taking logs of a product turns that product into a sum.
What does 'FCM' stand for?
@Dimitrij Celov: +1 for your time. I am sorry about my question. I fixed all the mistakes (parameters->variables) and giving more meaningful names to the variables. And sorry about not linking to my previous question (somehow I thought I did). I also clarified that this question is based on simulation and not on real-life empirical values. The values I am using are just values that are believed to be ok as there has been no real experiment. I am designing a protocol but before doing that, I wanted to understand the dynamics of my environment. I am still reading your post to digest everything.
+1, log scale sounds good to me -- @sprugman: couldn't you just preprocess your data and log-transform it before dispatching it to your charting routine?
I don't use R. I tried DBSCAN in R, and it was *much* slower, even without indexes. Anything that is not a matrix multiplication seems to be slow in R.
http://en.wikipedia.org/wiki/Margin_of_error#Calculations_assuming_random_sampling
@James I could be wrong but if the iterations are simply identifiers then most people will naturally report them in an increasing order like so: 22 24 36 44 ... The fact that the numbers are reported in an apparently random sequence suggests that each number represents the failed iteration number of 7 separate tests.
@psj I understand there are preferences on ways to display the results.  I did not realize however they force what you can and cannot report.  To force people to report results from a particularly model in some pre-specified format is not good practice, especially when the format is wrong.
Can you say a little bit more about how the data failed the chi-squared test?
Good response Michael.  Thanks much.
Bear in mind that each observation & prediction is binary, & their match / mismatch is a Bernoulli. So the variance / SD is a function of the mean.
+1 I'd only add that the Wikipedia page on [binomial proportion CI's](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval) has a fairly extensive discussion of various approximations for the small-$n$ case, including Agresti-Coull. Some people use the simple "add 2 successes and 2 failures" ... which I think is what Agresti-Coull gives with $z=2$.
Is that not just the same as asking to investigate non-linear relationships of predictors to the log-odds of response?
For completeness, could you provide a little more explanation as to why using an arbitrary scale invalidates the use of the CV? Thx!
Could you add this as the answer? Might make it easier for anyone else looking :-)
There was no waste of time.  We benefit by having a good question expressed in various ways, because that can enhance future search successes.  When we close a question as a duplicate, it usually remains on our site--it is rarely actually *deleted*--and thereby becomes part of a Web of links among related questions.  Finally, identifying the duplicates is one way we can get you a good answer really quickly. I am glad that it helped in your case.
Those look like numbers (I'd guess perhaps counts, though that distribution is rather odd), rather than categories.  Are you asking if they're maybe integer? You could easily check if any in the sample are not integers. (But who knows, maybe they're serial numbers or something, which could be categorical/ordered categorical.) .... but then why are you trying to model with variables if you don't know what they are?
If this is for some subject, could you add the [`self-study`](http://stats.stackexchange.com/tags/self-study/info) tag please?
Your question gets more interesting when $C=D$ (that is, you are using two different explanatory variables $A$ and $B$ to estimate a response $C$). (I believe the answer amounts to understanding how $L^1$ and $L^2$ norms differ.)
Can this your statement `it's a constant in a dimension at an angle to your original axes` be comparable with my `is exactly linearly dependent on the predictor(s)`, or you mean something different?
What do you think of modeling the estimated fixed effect as function of the time-invariant variables?
You are taking little excerpts that create gaps in our ability to understand what is going on.  You need to describe the model and the hierarchy of priors better for us to help (at least for me)>
(+1) Very nice, thanks for sharing!
While LARS and the lasso are closely related, for a fixed number of predictors, they may not even include the same variables. One could pick *a* penalty value for the lasso that gives the desired number of predictors, but the choice in neither case will be unique! Hence the OP has not yet provided a well-defined procedure, which is part of the problem. For LARS, there is the nice benefit that the penalty values yielding a certain number of predictors forms an interval, so picking an endpoint (which one?) or midpoint or some other criterion is somewhat easier.