tttjjj/elt_bench huggingface dataset:
Not enough detail here. What is your response (outcome) variable? Unequal group sizes (presumably what you mean) shouldn't derail ANOVA. What GLM did you use? Can you post the data? The answer may well be: use a GLM with appropriate family and link, but we can't possibly advise on what those are while being told *nothing* about the response.
+1 for samiam. This is the tool we used in the introductory lessons of the Stanford class on Probabilistic Graphical Models. It is nicely suited to viewing conditional probabilities and seeing how they change as different variables are observed.
@EngrStudent: Thank you for your reply, but it went way above my head what you are trying to explain esp terms like lower term. I can follow that the coefficients in front of the lagged innovation but what is the context of the mu here? I was hoping to get a reference to the link  for this model that I had found in Mathworks documentation but alas I forgot to write down the reference. Could you help in finding the reference for this model?
@sooprise Technically, the correct interpretation is more complicated than that. If anything, it makes more sense to compare the lower bound of one interval with the upper bound of the other. Since 78.8 is higher than 55.5 and there is a lot of overlap between the two intervals, you cannot rule out the possibility that rule B is in fact “stronger” than rule A.
Am not particularly active on CV.SE, so this may need a bit of re-tagging.
I looked at the sctest documentation, it doesn't even mention Chow test :) A general comment on dummy variable tests: in econometrics the datasets are usually not that big, and even the base model struggles to have enough observations to estimate the parameters. So when you add a dummy and all its interactions, often there's too many parameters to estimate, and most of them come not significant.
I edited the main question, but for C) something like that could work.
Are some of the items reverse coded? When you look at alpha, do some items have negative correlation?
Thanks Stephan....I see from your profile you are definitely the man to answer this question!  I will experiment with all of the above.
Maybe you should ask the moderators to move this to stats.SE.
Sargan test statistic is $n R^2$ for a certain auxiliary regression, and hence is guaranteed to be non-negative. You would want to check Hausman's original paper, which I think is very clearly written, and the `ivreg2` Stata Journal papers (http://www.stata-journal.com/article.html?article=st0030, http://www.stata-journal.com/article.html?article=st0030_3). I won't be able to demonstrate all the matrix derivations, but I expect them to be out there.
Regarding your link: Note that there is a big difference between being *positively* correlated and *perfectly* correlated! Regarding identical betas, you might think about two simple cases: What if $\\beta = 1$ and what if $\\beta = 0$? Also, your "definition" of beta is incorrect.
thank you for the answer!
A few more thoughts: I assume you have `ones` defined somewhere not in the visible code?  And is `C` large enough?  (It looks large enough to me, but that's a model-specific thing.)  I also note that it's possible for `mu` to equal zero or be negative, which would be bad, because of how `scale` is defined, and I wouldn't be surprised if the starting values for `beta0` and `beta1` were 0 since it's the prior mean.
Thanks Peter Flom. I will do the searching then. :-)
Alex, this is completely irrelevant to your question, but my original PhD was related to methodologies for integrating a diverse range of information from different sources (Geophysics, Geology, Petrophysics, Production & Reservoir Engineering data) in oil & gas field exploration & development. And what would I do now? Applications of ML & Information Theory in Financial Markets. A lot more difficult, challenging & better paid! :-)  Whatever you decide, just have FUN!
@grajlord in this particular case, yes, but you can't generalise beyond this to "tests" in general.  A lot of statistical tests depend on asymptotic properties and will be highly misleading with small samples.  t-tests happen to have been developed specifically for small samples.
Do you have the link to the manual? It helps if you supply the links to your citations in the post, fwiw.
Based on your suggestions, I'm guessing your data points are independent, and that the null hypothesis you want to test is of no association, i.e. a completely flat relationship. Linear regression with "robust" (i.e. heteroskedasticity-consistent) standard errors provides a Wald test for what you want, that is valid and efficient in large samples. It requires no assumption of true linearity.
"Check significance" implies a hypothesis. Do you have one regarding $(b+f)$? I can't think of anything that would make sense, so please clarify your intent.
@Alvaro Yes, you may invert sign of loadings of any factor to make it more pleasant for interpretation. But: if you compute factor scores don't forget to invert sign there as well; with oblique rotation also don't forget to change sign accordingly in the structure matrix and in the correlation-between-factors matrix - if you're going to report these results.
The above approach can work, but will be very inefficient as the data matrix will be very sparse. Just think about how many zeroes you will have for 'Spanish'. What can be done to improve things is to code some of the items into numerical variables (carat and weight for example) and others into categorical variables (material and type of bijoux for example). While this would require more data processing work, it would provide more stable and ultimately useful inferences (and you would be able to use interaction effects as well).
@Glen_b, as I can feely choose which which sets to actually use for A and B, I'd be happy as long as I can do either direction :)
@Cardinal: Perhaps Dail can clarify, but I took it to mean he wanted to test the mean requirement of weak stationarity, which claims that if $X(t)$ is our time series, then $E(X(t)) = \\mu, \\forall t$. Undergraduate courses typically test this condition by fitting a linear regression through the series and testing if the slope is zero, but this approach is not valid if the data are not normal. Could you clarify that if this is the case, Dail?
The partitioning of X leads to 4 pieces in X'X, and two in X'y. The aim is to keep those partitions, because you want to compare each piece the two different ways that are supposed to be equal. That leaves you with two parts in X'X and one in X'y that needs to be shown the same in both (the other parts are obvious).
It depends on what you mean by "analytic formula." The expectation is an [exponential integral](http://en.wikipedia.org/wiki/Exponential_integral).
The idea is to use the definition of conditional density $f(y_1\\vert y_2=a)=\\dfrac{f_{Y_1,Y_2}(y_1,a)}{f_{Y_2}(a)}$. You know that the joint $f_{Y_1,Y_2}$ is a bivariate normal and that the marginal $f_{Y_2}$ is a normal then you just have to replace the values and do the unpleasant algebra. These [notes](http://www.stats.ox.ac.uk/~steffen/teaching/bs2HT9/gauss.pdf) might be of some help. [Here](http://fourier.eng.hmc.edu/e161/lectures/gaussianprocess/node7.html) is the full proof.
No, not unless you're prepared to defend why the significance of this one pairwise comparison (out of 10 such possible comparisons) is greater than would be expected by chance alone.
Note, that he sorted `y`. It does not make the regression model appropriate, though.
No, that doesn't seem right. Suppose we have many $c_i = \\epsilon$, and then some more expensive ones: a greedy sampling procedure would buy -all- of the cheap bits before the expensive ones, but there is no reason to buy all of the cheap bits -- we quickly have high accuracy there.
Glad it's working well for you!
I know how to do the equivalence between ridge regression formulations (since it is differenciable) but I'm wondering if it would be any easier to consider only the $l_1$ norm instead of the elastic net penalty?
It would be helpful if you clarified your question with additional information and more detail about what you're trying to achieve.
It's clear that the tolerance used by the R routine is fairly loose in comparison with MATLAB, but in both cases the norm of the residuals is essentially 0.
+1 (Your answer came in while I was drawing my figure...)
I am really not sure which is the boosting algorithm thats default to the starts group node, but there is another node in SAS EM called Gradient Boosting (which is not giving any encouraging results). The point that I want to make is, I run this boosting technique (not the Gradient one) on my dataset without any adjustment of priors/resampling/profit loss matrix for 5 iterations (no. of loops) and the result gave me around 20+ business rules/patterns.
@Russ Lenth Yes they are known.
One way would be to find the average proportion of letters that come after your letter; e.g. ABABC  - 4/5 come after first A and 2/5 come after 2nd A, average is 3/5. Then that could be the DV in a logistic regression.
Are the Clicks also in the Visits (i.e. Clicks / Visits can never be greater than 1)?
Ok got it , then yes.
@PeterFlom could you help me reword the question? I am not proficient enough in this field :)
This seems like a very open-ended question and the best approach might depend on the research goal. Do you have a particular one in mind?
That may be, but it doesnt make sense logically to include only some of the indicators and not others.  Choice of coding might also influence results then! The only logical solution is to use software that treats factors correctly.
Perhaps you should explain your reasoning.
As can R (the `rlm` method in the `MASS` package), & SAS (`PROC ROBUSTREG`).
@Frank  You make a good point.  Ultimately, the goal for this project is to predict outcome of further events.  (So, I guess it can be thought of as a machine learning tast with training data.)  Some outcomes are more "important" than others, so I was looking for a way to weight them accordingly.  Nick's suggestion for the likelihood function makes sense and should be fairly trivial to implement in code.
If I'm right you mix up Viterbi training and Viterbi decoding.
Thank you. I found the same thing for #1 - I believe that Will Potts paper had a typo and that the calculation of the survivorship function is as shown in Singer and Willett. For #2) I was asking about the mean, not the median. I later found explicit calculation of this in Hosmer and Lemeshow where they did exactly as I show - i.e. including the "1" of S(0). This is at odds with answers I received in the linked questions. Which leads me to believe that my #3 question above (MRL) is as I show here and not in the linked answer.....
@ Michael: U can find the results [here](https://docs.google.com/document/pub?id=1_4sDxvwFLY0-SBvW8llkURv23My89Vc06ZNbepCcHAw).
Divisive: a film that everyone rates at three stars is not divisive, but a film that half of people rate as 1 star and half of people rate as 5 star is divisive. (Basically, the larger the standard deviation, the more divisive the film.)
+1 I'm surprised any simulations at all were needed--they are not.  Such rules are easy to figure out from general principles.
Exactly how do `vgm` or `fit.variogram` not "work"?  What happens, exactly?
+1 For emphasizing the need to frame a question.
comment from Iee (transferred from message): I don't know that much about stats (the discipline)...I get that I will have to calculate for each possible cluster number some stats and compare the results. But what are the relevant statistics here, for instance with the elbow method it is the percentage variance explained, but how would I calculate that (a high level algorithm for the easiest technique that will get the job done would be massively appreciated!).
I will edit the answer
Thanks @Dimitris for your answer. I need concrete derivation of this problem for my better understanding.
@ars Here it is (as Gist): http://gist.github.com/629642, http://gist.github.com/629644, http://gist.github.com/629645.
Glad that you find my answer helpful.
Mathematically speaking, in binary classification, knowing the worst is equivalent to knowing the best. If you know the best you reverse the rule and it becomes the worst. Why do you want the worst rule ? what do you mean by worst ?
Much easier to interpret coefficient that way
It might help to know a bit more about your data, what is the response variable and what are the covariates?
The question was from ML.SE which is closed now - apparently the admins hadn't merged in all the questions. I edited the text now to explicitly state the problem!
Why not multiply everything by -1, then proceed as usual?
I imagine you could get such research published in a journal related to human resources or something similar. And I would be very surprised if such a field had a lot of experiments or random samples from well-defined populations.
If I recall correctly, isn't the log-normal distribution not uniquely defined by its moments.
Hi, thanks for reply. I'm trying to code Optimizer myself to understand it well. I wrote BFGS optimization algorithm in Java and use Wolfe Conditions for Line search and it is producing global minimum. Used Rosenbrock function as my test fixture. But BFGS I wrote is unconstrained optimizer. If I want to use BFGS algo for GARCH MLE then I need to modify it to constrained optimization algorithm to satisfy MLE constraints. But before going ahead I want to know whether constrained BFGS algo is better suited to solve GARCH MLE like problems. Thanks for your time.
No one said ESL was about time series...
I have read that paper.  It is not convincing, and Liu has not studied a variety of the kinds of clinical trial situations I described.  More arguments are at http://biostat.mc.vanderbilt.edu/wiki/pub/Main/RmS/course2.pdf in the chapter about analysis of serial (longitudinal) data.
My hunch is that you might be able to repeatedly apply Bayes theorem, starting with an uninformative Beta prior (see [this answer](http://stats.stackexchange.com/a/44215/25936) for an example of a single application in the Beta/Binomial model). Then you would effectively be multiplying your MLEs together and incorporating the uncertainty directly. Would be interesting to know more about your algorithm to see if this is applicable.
Well, no, it really doesn't have an interpretation as a percentage. Your comment adds nothing to your question; please don't do that.
Please don't [cross-post](http://stats.stackexchange.com/questions/68344/two-level-inputs-and-output-in-fuzzy-matlab-implementation), this didn't work out & is explicitly discouraged by SE policy.
@user603 Can you explain "You either have to acknowledge that a series is highly correlated with it's own past by putting each y1,t as a dimension of it's own (i.e. resulting in N*T dimensions)" please?
It would be good to consider accepting answers to some of your previous questions, all of which have received multiple answers, thus giving you some choice. There is a check mark next to each answer that you can click on to indicate which one has been addressed your query.
@NickStauner very informative link. But I must say that, considering the discipline (software engineering), I am already delighted that they bothered to do a factor analysis and calculate reliability at all. I can't imagine requiring them to address such issues before the article is accepted. Especially with their scale, which doesn't even claim to be generalizable (it is specific to their software product).
Clearly you have multiple students for each teacher.  Do you have multiple teachers for each student?
interesting, +1. Can you suggest an introduction t deconvolution methods ? Would you have any comments on fitting smooth f to not much data in 5d or so ?
I would suggest that you precise the form of what is called an exponential familly here. That will bring pedagogy to this question (which is "scolar") and also will tell wether it is in a canonical form or not.
if your predictions are plain straight lines, you're probably looking for linear regression: http://en.wikipedia.org/wiki/Linear_regression (perhaps with additional increasing weights for later events)
There's no r in *nominal*.
One possibility -- if you have access to zero-inflated beta regression, you could try to model how much they *don't* send back.
If this is homework please add homework tag.
I confess I am unable to see how any of Tukey's EDA methods would identify changepoints for these data.  Which one(s) were you referring to?
@matt b, I just found an ungated version of the [paper](http://www.mrc-bsu.cam.ac.uk/personal/chris/papers/denstrip.pdf). Also check out the R package [denstrip](http://cran.r-project.org/web/packages/denstrip/index.html). In retrospect though I think the suggestion may be overkill, you will only have slight gains over just plotting the quantiles of interest and it will be much more computationally expensive (hence reducing its utility as an exploratory tool).
Many (but not all!) of the properties and usefulness of Shannon entropy arise from its convexity. If you look at the build up of the basics results in information theory, they more or less hinge on Jensen's inequality. So, in a certain (vague) sense, there is not too much that is (terribly) special about $-\\log x$ as the particular nonlinearity that leads to a notion of "information".
I suspect you're seeing this because of some low cell counts (some books I've read suggest a min. of 5 per cell). Maybe someone more knowledgeable on the subject can chip in?
@StephanKolassa I designed this problem by myself to learn bootstrap. The definition of $X$ is arbitrary. My goal was to get outcomes of a unknown distribution to use the bootstrap method.
Winning is not necessarily transitive.  Even in games of chance that is not the case: see [Efron's intransitive dice](http://en.wikipedia.org/wiki/Nontransitive_dice).  These ranking systems adopt particular models of winning but they are not appropriate for all forms of competition!  (This is particularly true where contests are decided by a voting procedure.)  Thus it's possible that your 80%, 80% numbers would be consistent with *any* winning probability for C against A.
Thanks for the detailed response! I have read some of the provided references, but will definitely take a look at the others. As much as I understand the evil of p-values, reviewers unfortunately often think otherwise (at least for now). As recommended by Bates, I am using mcmc sampling, which from my understanding sidesteps part of the problem (i.e. that it is not possible to correctly estimate degrees of freedom for an lmer model). The characteristics are a DV, I will add some more information to clarify.
Why don't you just consider an algorithm that *guarantees* newly generated random values are not in the set? There are plenty of efficient ways to do this. For instance, since you're worried about the set filling, you have enough storage for all $n=10^5$ elements. Why not represent the set as an array of indexes $1$..$n$ along with the current set size $k$? The last $k$ entries in the array hold the set's indexes and the remaining entries hold their complement. To insert a new element, swap a randomly chosen element from the first $k$ with the entry in position $k$ and decrement $k$.
Reducing the overlap with `unique` or by rounding can result in biased (deceptive) plots.  It's important to somehow indicate the amount of overlap through some graphical means such as lightness or with sunflower plots.
I've voted to close, it is really unclear what you are asking for. Try to be more explicit of what your question is.
I also vote for Kruschke's book. I've browsed most of the books listed in the answers and this is the one I found most clear. IMO, it is the most clear stats book I have read. It helps a lot that R code is available to match formulas with code. The author starts with very simple examples and builds on them. Very little background is needed. All reviews on Amazon are highly favorable. Hoff's book is my second favorite.
Thank you so much, I've got an error when importing data into R corrected with 
`time <- c(rep(0,5), rep(2,5), rep(5,5), rep(10,5), rep(15,5), rep(30,5));
sample <- c(rep(1:5, 6))` so the R transposed table was bad; now I've got the same results.
if you use cd algorithm with lasso you can select more than n variables
The Wikipedia article on [Support Vector Machines](http://en.wikipedia.org/wiki/Support_vector_machine) answers this by pointing out that the nonzero Lagrange multipliers correspond to points on the margin, of which ordinarily there would be very few.
Although I believe most readers would have some idea of what "bin-smoothing" might be, it's a broad technique and conceivably could be applied in several distinct ways.  Could you please tell us more precisely what *you* mean by it in the context of regression analysis? Could you supply a link to a reference?
Thank you.  Could you also tell me how these scores (resembling probabilities) are calculated?
30% goes to 40%....
Hi there, welcome to the site. Did you mean to say "negative decline in X", which I read as meaning "an increase in X"? Generally, a negative `X` coefficient means that an increase in `Y` is associated with a decrease in `X` (and vice versa).
You can be almost sure :)
Rather than "no multicolinearity" I'd say "no linear dependence". Collinearity is often used as a continuous rather than categorical measure. It is only strict or exact collinearity that is forbidden.
@Wayne If you can direct me how to do it we can go to a chat room for more detailed discussion. I have no clue as to how to do that.
Anyone with any help?
There are also several threads [discussing how normality is essentially irrelevant](http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless) for many purposes. If you have independent observations, and at least moderate sample size, the only thing that matters for OLS inference is that all the residuals have the same variance. Not Normality. If you use robust/heteroskedasticity-consistent/sandwich/Huber-Eicker-White estimates of standard error then even the constant variance requirement isn't required.
@JeremyMiles thanks for the recommandation to write all teh codes..the packagef or pcr is library(pls). and althought pcr does principle component regression , it uses PCA and in the output of pcr we can see the percentage of explained variance taht each principle component explains. I want to use these codes on a high dimensional data and I wanted to check which code gives the correct value.but none of these codes on a simple example didnt work well.so I dont know what the problem is!and which one can I use..
How would such a determination help you with your analysis?  One good answer, found in many places on this site, is *not at all*.  Indeed, this (Stevens') classification of variables has been criticized (justifiably, IMHO) for unnecessarily *limiting* the analyst's options.  This suggests you might get more value from this site by forgetting about variable type altogether and asking the questions that I suspect may be lurking beneath the surface here, questions concerning how to understand and analyze your data.
Is your problem to model the pct% of hours contributed by each position? You might want to try a logratio transform--see http://en.wikipedia.org/wiki/Compositional_data
The Wald approximation can be wildly inaccurate where the proportions are close to zero or one, as they seem to be in this case, and is not a recommended method in the paper that you link. It should probably never be used. See this: http://stats.stackexchange.com/questions/15567/putting-a-confidence-interval-on-the-mean-of-a-very-rare-event/15568#15568