tttjjj/elt_bench huggingface dataset:
install.packages("lattice") and library(lattice). If you use `SAS` you can get both of these plots there, as well. Let me know if that's the case and I will provide more details.
(+1) The first paragraph is very well put. About point 2, I somehow missed this article, so thanks for sharing! Of note, correlations of comparable magnitude would be observed with traditional factor analysis models (where loadings mimic the discrimination parameter -- under the CTT framework, it would be the point-biserial correlation coefficient) provided item difficulties are uniformly distributed on the latent trait (i.e., no too difficult nor too easy items).
http://en.wikipedia.org/wiki/Absolute_value#Definition_and_properties
Can you say more about your situation? I can't really follow this. Can you add your contingency table & the noise data?
(1) There are two samples, not three.  (2) To which p-value do you refer: the t-test or the MW test?
1.) "10,000" is too small for your question--try "250,000" (or more). 2.) If you run a Monte Carlo simulation and get a result that seems strange, try changing the seed with `set.seed`. 3.) Don't always trust Wikipedia--note how your quoted text (from the "Median" article) differs from this *other* Wikipedia [article](http://en.wikipedia.org/wiki/Bias_of_an_estimator#Bias_with_respect_to_other_loss_functions) 4.) Your **R** code is a total mess--check out [Google's R Style Guide](https://google-styleguide.googlecode.com/svn/trunk/Rguide.xml) for some good style guidelines.
and if you google for matlab hurst exponent you'll find several.
You will have to correct a few square-roots, but apart from that, there's nothing wrong with it, since :

$$E\\left[(X_i-\\mu)\\frac{sign\\{X_i-\\theta\\}}{2f(\\theta)}\\right] = \\frac 1{2f(\\theta)}E\\left[(X_i-\\mu)sign\\{X_i-\\theta\\}\\right] = \\frac 1{2f(\\theta)}E[(X_i\\cdot sign\\{X_i-\\theta\\}] = $$

$$=\\frac 1{2f(\\theta)}\\cdot \\Big(E(X_i; X_i> \\theta) - E(X_i; X_i< \\theta)\\Big)$$

$$=\\frac 1{2f(\\theta)}\\cdot \\Big(\\int_{\\theta}^{\\infty}xf(x)dx - \\int_{-\\infty}^{\\theta}xf(x)dx\\Big)$$

$$=\\frac 1{2f(\\theta)}\\cdot \\Big(2\\int_{\\theta}^{\\infty}xf(x)dx - \\mu\\Big)$$
I managed to overlook that part somehow. Sorry!
Sorry for the late response! Thank you. I just thought using the joint distribution was very weird. In regards to the second question, what expression would you use for $\\Sigma^{-1}$ when writing the trace explicitly?
Yes, but the variance term of the normal distribution is equivalent to a change of base of the exponent.  Even the way you've drawn it, the base could be $(\\sqrt{e})^{-x^2}$
I believe that the model is spatially auto-regressive, so the appearance of $y$ on RHS is a spatially-weighted mean of the *neighbors* of the observational element in $y$ on the LHS.
I meant that you have scored subsets of the set of all elements. Some form of binary regression will assign scores to the individual elements. The form of regression depends on how you model the score of a set depending on its elements. You cannot build a unique hierarchy of sets, since you stated some overlapping sets are not in a hierarchical relationship. You could have an inclusion graph, but that's not a partition and has little to do with clustering.
"*" simplify means multiplication.
The function being fitted is not linear. The default link function is logisitic. That is $$P(Y=1) = \\frac{\\exp(\\eta)}{1+\\exp(\\eta)}$$ where $\\eta = X\\beta$. You really shouldn't use this model, or any other GLM - or indeed any model - until you at least have [a basic understanding](http://en.wikipedia.org/wiki/Generalized_linear_model#Intuition) of what it is you're fitting. There are some good basic introductions to GLMs around here and there.
Many mapping software platforms have the capabilities mentioned in this post. The distortions based on attributes when it comes to maps are frequently referred to as cartograms. See http://gis.stackexchange.com/q/7406/751 . That being said, bars placed happenstance in a map aren't any easier to visualize than colors. When the bars aren't side by side they are difficult to make relative comparisons, which isn't as big a problem with a color scale.
You're right @jcb, there's some sloppiness here, but note that $X$ has $n$ elements.  Thus the presence of $k$ in the upper limit suggests $k$ depends on the cluster, $i$.  (Therefore the two sums should be interchanged...).  BTW, I converted your reply to a comment because although it makes a helpful point, it doesn't answer the question.  Perhaps you would care to make a direct answer?  Regardless, thank you for your contribution and welcome to our site!
A confidence region for what?  The mean vector?  It would be an ellipsoid involving the inverse of the sample covariance matrix.
I like this. I'll look further into it. Thank you.
What do you mean by application of cAIC or mAIC "outside of AIC"? DIC is a widely-used measure of predictive accuracy that you could investigate, that tries to penalize by the "effective" number of parameters included in the multilevel model.
Thank you, will check this one!
You should probably spell out HMM and CRF; acronyms can be tricky, especially for people who are not native English speakers.
"*If I want to omit the constant term, then you mentioned that I need to omit the mean(y)*"  -- no, that's not what I said. Omitting the constant term corresponds to a model where the mean is zero  (at least if $|a|<1$). If you're generating data from the deterministic model you mention, AR (or regression more broadly) really isn't appropriate. If you're interested in AR models, don't use the logistic map to generate noise, but if you're interested in modelling that deterministic process, don't use AR for it. Its assumptions aren't satisfied.
@Glen_b Apologies for the late reply, I've been overseas (and still am). The difficulty is I don't know the algebraic manipulations from the equation $\\frac{1}{p}\\left[Y - \\frac{Y^2}{2} + \\frac{Y^3}{3} - \\frac{Y^4}{4} + \\frac{Y^5}{5} - \\cdots\\right]$ onwards. That is, which terms do I drop and how do I expand things so I am left with all 4 expressions: sample mean, sample variance, sample skewness and sample kurtosis in the final expression.
Thanks for your answer, I considered your suggestions. I feel, that for my problem, just weighting the score (w/o missing values) with the number of actual matches works quite well..
The url had changed. I've now fixed it.
I do not understand your question. Yes, k-means uses means, i.e. least squares estimators. If your data is generated by k distributions that have the same variance and are well separated, k-means will likely converge to the best (as in: **least squares**) estimation of the true cluster centers. In reality, your data will however not be this simple.
@StephanKolassa Done. You're right it's a dichotomous variable.
Why do you have to lag the variables three times? Is that the same as to lag the first differences two times?
Extreme is difficult to judge, every function approaches linear over some range; as we know from the Taylor Series decomposition. Why wouldn't Burnham and Anderson's information criterion approach to model selection  serve this problem well?
thanks, +1 for the solution
Thanks for pointing out the `penalized` package.
Note that the Hosmer-Lemeshow GoF test has been shown to depend on the binning used / be unreliable.
Can you please send me the data (my e-mail is easily googlable ...)?  I would very much like to see if I can see what the problem is.
Could you explain why you are advocating such an approach over using, e.g., the built-in `R` functions such as `pnorm`? It also seems unclear whether a programming solution is what the OP is searching for over, say, an explicit expression.
I don't think either method "takes into account" the value in a  precious interval. Furthermore, you should only use 5 dummies if you have 6 levels, that is only if you choose to do it the hard way in not using a factor variable. You could always create a cumulative dose variable.
@Zhang: 1. decision trees are also based "on counting" and calculating approximate probabilities (similar to bayes), so this is no argument. The question is whether you want to have rules at the end (as stated in one of your comments) or not. 2. If you do not want to have rules, I'd try a naive bayes first and if the results are not satisfying, switch to bayesian network 3. a) Bayesian Networks do take continuous variables, it is just more difficult b) finding the correct/best network architecture is an NP-problem. I suggest to learn/read more how to build BN.
Thanks a lot Nick! You helped me a lot!
There are other discussions about the R-squared when there's no interecept, such as http://stats.stackexchange.com/questions/47527/use-squared-correlation-in-regression-without-intercept Did you take a look at these discussions too ?
I'm only suggesting using a log scale for the x-axis. I don't think a log scale for the frequency axis would be a good idea, as then the shaded area of each bar of the histogram wouldn't be proportional to the number of observations.
Thank you for your comment and for your edit. I figured a Bayesian or machine learning approach might be best. Unfortunately I am relatively unfamiliar with these approaches.
(Years later), there are many many ways to tinker with MFCC;
Kinunnen et al.,
[Frequency Warping and Robust Speaker Verifications: A Comparison of Alternative Mel-Scale Representations](http://cs.joensuu.fi/pages/tkinnu/webpage/pdf/melwarp_IS2013.pdf)
2013, 5p, use 60 coefficients.
And, optimize what ? On what non-open database ?
So I'd say (non-expert) that the question is too broad to be answerable.
"Best" in what sense, exactly? After all, there are infinitely many ways to create categories. What is the purpose of grouping them?
@whuber that's an excellent point, I've added an addendum to my answer to reflect your insight.
Additionally you can use temporary arrays in sas data steps in a similar way to how you would use matrices in R.
Is the outcome binary, aware or not?
Although it's a good statistical question, if I were at the receiving end of this I would likely flag you as a spammer after the end of the first couple of weeks and discontinue any relationship with you. If there's any chance your new customers could perceive these contacts as negatively as I do, you might consider rolling out this program with a small sample of the population first.
Thank you for the response. I had initially used linear regression to examine this, with which I tested the change in mean between tow time points. However, the challenge comes when testing for the general trend over all the time periods. That means having one significant (or non-significant) value. I believe there are several assumptions that have to be incorporated. I have also seen that non-parametric tests (Mann-Kendall) can be used to test for trend. I am not quite sure how feasible this will be with the data I have maybe for non-normal data.Will look at the packages suggested. Thanks.
'diagnose' is a verb. From the role the word is playing in your sentence, I think you need a noun there. Would 'diagnosis' fit what your variable contains? If you have three different response variables, you may be looking at MANOVA. If they're close to independent it may be reasonable to look at them separately. As to whether you do a correction for multiple testing depends on how much you worry about type I errors as against type II errors. How you view this may also depend on your audience.
Also: if that list is public, mind sharing? Would be a nice resource to have :)
Welcome to the site. With your couple of upvotes from this post you should have enough reputation to post as many images and links in questions/answers as you would like.
My pleasure, kiddo.
I don't see the problem here. You can't take the difference of two data sets, only the difference between a function (like the mean) of one and the same function of the other. If there's some reason to disapprove of the mean (which I don't understand yet), there are other options (like the median). The standard error of the mean difference is the denominator of the t-test, which you can find anywhere. Can you state your final goal & why you don't want to work w/ the difference in means / t-test?
I know this problem as [link prediction](http://scholar.google.com/scholar?hl=en&q=link+prediction&btnG=&as_sdt=1%2C5&as_sdtp=). It's a novel subject in the semantic web domain. I think people are just beginning to investigate machine learning techniques.
In principle, just by knowing $f(x,y)$, you implicitly know the distributions of $X$ and $Y$ including their expectations and also $\\E(X \\mid Y)$. So, all of the latter "extra" information is superfluous. Note that the transformation $(X,Y) \\mapsto (X,X+Y)$ is one-to-one and so transformation-of-variables will give a density for $(X,X+Y)$ (which is easy to write in terms of $f$). From this, you can then easily recover a simple formula for $\\E(X \\mid X+Y)$. But, saying anything more explicit within the very general assumptions you've made doesn't seem likely.
thanks for the links, I will try and go through them as much as I can in the coming weeks... I have been exposed to R once before, in survival analysis course in which we did a lot of multivariate regression (cox and aelen models) and a bunch of other stuff I can't really recall. My impression of R, as a person who's very used to MATLAB was quite negative, but I did it had a lot to do with the fact that we were more or less thrown to the deep end of the pool, and then expected to learn to swim on our own, which of course led to me hating the software from then on :) Time to change that perhaps
Douglas: Because I paid more attention to your answer :-).  Please don't get me wrong--I like it and I like this thread. I thought it important to point out that you had to add an assumption in order to obtain your answer, that's all.  As a practical matter, in many situations--including this one--*there is no prior*.  (I sure wouldn't want to make up a personal prior and then have to bet big money on it!)  But of course there is still an optimum, *provided you specify a loss function.*  ("Maximizing" an expectation is not a full loss function.)
Hi @TomReilly I have tried to post the data for the month but it is too long please advise how I can send this to you to look at?
Ah, that makes  sense. I don't have the data on hand, but I agree with your point about the behavioral measure; the reviewers did mention some concerns. Thanks for the assistance--when I get a hold of the data I'll follow up.
Thanks. It worked - this way I can plot both the Class random effects (level=1), and the student (level=2).
This question was asked in Mathematics: [About joint probability divided by the product of the probabilities](http://math.stackexchange.com/q/79203/29951).
@blerta re: plotting multiple ROC curves with pROC; try adding add = TRUE to your plot.roc() command, to all but the first ROC you try to plot.
So, I solved this problem by using counts, however this was unique to my case in which the distribution of counts for the separate ordinal responses were approximatley poisson distributed. However, the problem remains in the case where they are not, perhaps summing up the categorical levels could yield results (though at the loss of information).
I would like to add blackjack as well.
Really excellent notes! Very witty too: "Many financial time series exhibit an exponential trend, not always in the right direction for the owners of the corresponding assets."
+1 as usual, comprehensive, clear and helpful.
Not correct in general.  If the slope of baseline weight is not 1.0 then analysis of change will not be equivalent to analysis of final weight unless initial weight is in both models and you are using ordinary regression.  If baseline weight is in two places the model is actually more difficult to explain, so reasons for persisting with this approach are unclear.
Not sure if this is useful, but survival/event history/time before failure models are used to answer the dual questions of *whether and when an event occurs*. By asking about *duration* are you trying to answer *whether and when a task is completed*? If so  a survival/event history/time before failure model might be useful.
How were the numbers of measurements chosen? This might bear on your choice of statistical analysis.  What is "i" in your table? Does it identify different patients, different parameters, or perhaps different sets of measurements of the same patient?
Thank you  for explaining the patterns, Eric.  Your comment regarding the duplicate points prompted me to create a density plot.  It looks as if the cloud has a certain bias towards the 'good classifier' neighbourhood. Are you familiar with a way to quantify such a bias and to say whether it's statistically significant?
Got it, thank you for your answer. The part that hadn't clicked before was #[elements being sampled] + #[bars] = n + n - 1 = 2n - 1, so that we can think of 2n-1 locations, of which we select n-1 for bars (or, equivalently, n for elements).
Agree with user603. Please check page 21 of http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf. This is based on R but includes a good discussion of the theory behind linear regression.
Think of this from a CV perspective. Let us say, we are doing a 5-fold CV. Under method 1, you are measuring out-of-sample performance of the trained rule not only on the different set of samples, but also on the varying shift and scale parameters. Under method 2, you are measuring out-of-sample performance of the trained rule only on the different set of samples as your shift and scale parameters are fixed. I don't know if I have explained it properly. Do let me know if the above seems confusing.
I advise this to be a community wiki.
I made a presentation on linear and nonlinear methods for dimensionality reduction once: http://www.pardisnoorzad.com/wp-content/uploads/slides/High-dimensional%20data%20and%20dimensionality%20reduction.pdf
Part of this is addressed at http://stats.stackexchange.com/questions/89376/do-i-have-to-learn-sas-if-i-want-to-go-into-industry and indeed elsewhere here. It's difficult to summarize impartially, but note that many people challenge the alleged importance or dominance of SAS, with a mix of reasons varying from disliking it on principle to emphasising the usefulness and widespread use of other software in practice.
An other question, do the data need to be positive or normalized, because the representation of no-path can be many values, like "inf" or a negative value.
I am baffled by your last comment, Histelheim: what could "entropy without the logarithm" possibly refer to?  That suggests you haven't yet clearly articulated your question, because it sounds like you have some unstated concept of "entropy" in mind.  Please don't keep us guessing--edit your question so that your readers can provide the kinds of answers you are looking for.
It doesn't work the way you seem to think. You're confusing a bunch of different things together.
Kino, could you please clarify what you mean by populations to be "similar"?  Otherwise you leave the interpretation of this question to the reader, most of whom might not read your mind correctly.
I agree, if you take only that phrase. However, the context in which I wrote that sentence, I believe, defines that what I mean by 'correct' is in no contradiction with bayesian analysis in general. Basically, I'm asking for a measure that aims to 'approximate consistency'. The usual measures (e.g., bayes factors) are known to be consistent (i.e., they converge to the 'true' model if the true model is in the set of competing models).
Is a difference in ranks much simpler to interprete as Spearman's rho? Even if so, would you call Spearman's rho wrong? Sad that we don't see the reviewers reasoning.
In a way, yes. One way to understand kernel density estimation is that you approximate the density of a point from some distribution as a weighted average of its similarities with a set of points from the distribution. So the notion of similarity does play a role here as well.
@Glen_b I believe that this paper of Vuong (1989), http://www.jstor.org/discover/10.2307/1912557  provides a general framework for non-nested models.
It is rather difficult to get such a small SD with data like this: for a mean of 2.8, the SD has to be *at least* $\\sqrt{0.2\\times 0.8}=0.4$.  (Even if 2.8 represents a rounded value, the SD must still exceed 0.357.)  An SD of 0.54 implies that no more than two people could have answered with a 5 (with 21 2's and 62 3's) and not more than six could have answered with a 1 (with 5 2's and 74 3's).  This suggests the question may provide exceptionally little information because the scale does not effectively discriminate.
I wish we could move a question to Stack Overflow the way you can move a question from Super User...
`summary(m0)$r.squared` works for me, not name subscripting...
@StasK Thanks for the comments.I understand the point you are trying to make.
I was able to read the article at thefreelibrary.com. Not sure what kind of page that is though.
You didn't put it clear what you're after. Do you want to see the cloud, every data point? Do you want to see all the bivariate facets _at once_?
@Joël, what is the value of y[6283]? have you made sure that the values of the alphas and betas are restricted to legal values?  I expect that something may have gone to 0 or below and that gives the error.
*(continued)* If anything, in small samples I'd prefer *fewer* scale items, from a purely statistical perspective. With coarser measurements, there is less variation in the data to be explained. This might sound undesirable, but if you have very little data it contains very little explanatory power, so coarser measurements are less of a burden on the data.
"It follows that the parameters $\\gamma$ and $\\delta$ determine the shape of the distribution of $x$, that $\\lambda$ is a scale factor and $\\xi$ a location factor. It follows also that attention should be concentrated on the relation between the values of $\\gamma$ and $\\delta$ and the distribution of $x$, since the parameters $\\xi$ and $\\lambda$ affect the distribution only in a simple manner." pp. 152, same citation as above.
(I'll keep the answer, because especially with Bernd's comment, it may be useful to others asking this same question.)
Though I wonder whether the `lsmeans` package together with the `pbkrtest` package could provide good confidence intervals.
Your method does not generate a uniformly distributed vector on the simplex. To do what you want correctly, the most straightforward way is to generate $n$ iid $\\mathrm{Exp}(1)$ random variables and then normalize them by their sum. You could try to do it by finding some other method to draw only $n-1$ variates directly, but I have my doubts regarding the efficiency tradeoff since $\\mathrm{Exp}(1)$ variates can be *very* efficiently generated from $U(0,1)$ variates.
again, I don't know if yours is the correct Doane's formula, if you look at the paper for example he uses log2 instead of log
Both those tests are based on regression and should be equivalent.
+1, this is good, but to be honest, I like your "penalizing" suggestion better.  I can imagine people who wouldn't understand what you mean by "elaborate".
david why don't you redo the example normalising all the variables x,x^2, ...,x^n to zero mean and standard deviation 1, and then see the results you get... its not very surprising that your coefficients are small when your input variables are large
@ru_di Strictly speaking PPV are not random variables, so there’s no test to do, the higher the better...

However in practice PPVs are estimated using finite samples and comparison with "the truth" (or with some "gold standard"). In this case, the PPV is the proportion of people truly diseased among those for which the test was positive. 

I think you can just use usual methods for comparison on proportions, although the  number of people with a positive test is not fixed beforehand.

If both PPVs are estimated on the same sample, it is slightly different, I need to think it better :)