tttjjj/elt_bench huggingface dataset:
THANK YOU! This has been most useful. Thank you very much! I shall print a copy of this and let him process it in.
+1 This expresses some of the thoughts I had when I made that comment on GIS.SE. To go further, one can include data from a streets dataset which would (in effect) give strong information about what each GPS track is "trying" to approximate. Note that the *images* of GPS tracks on a map often are not smooth curves, but the tracks themselves *as functions of time* are smooth. Thus the timestamp (that automatically comes with GPS data, at least originally) is of crucial value in any analysis: that's one key way in which this is not exactly a "point cloud" fitting problem.
If you do an omnibus test first on all three means (ANOVA), you don't need to adjust for multiple comparisons in the specific case of 3 groups. This is called a protected Fisher's LSD, and the Type I error rate is maintained at .05 without further adjustment.
ZINB is useful when there are a lot of 0's.
The general, always working way is mentioned [here](http://stats.stackexchange.com/a/12925/3277). A particular way exploits the fact that Sperman rho is Pearson r mathematically (only for ranks), hence yo can convert it into euclidean d as explained [here](http://stats.stackexchange.com/a/36158/3277).
perhaps a bit kludgy, but you could always collapse the columns together in a new column and then use `grepl` or the likes on the new column: `apply(data, 1, function(x) paste(x[1], x[2], x[3], x[4], sep = ""))`
+1 good example, thanks
@JEquihua thanks. I will read the paper soon. Looks interesting.
This is a great post - thanks for making it.
2/2 Also, I think it matters how you define default. There is a huge difference between missing x payments (where some amount of the loan is recovered) versus not paying any portion of the loan at all.
I realize its not part of your question, but judging from some other clumsy phrasing in the article (where what was said may have been not quite correct but where the intent is easily inferred in context), I'd be inclined to assume they meant 90% intervals. Certainly if they were really to use 10% intervals, it would require some kind of comment as to *why*; the absence of such comment reinforces my feeling that it's merely clumsy/misleading phrasing. One assumes the referees include nobody with enough statistical knowledge to insist they either fix it or explain why 10% intervals are used
Formatting usually leads to a better reception of the question -- please, do it in your own interest.
Thanks. I handle skipped answers as in (c). Still, I wonder whether  I can test the equal of proportion of correct answers for group A and B, using a z-test (prop.test in R)? How is it different from t-test?
D'oh. I got it. The color of acceptance is really bad for be, because I am colorblind :) It's much better on SO, maybe CV should change it because there a quite a few colorblind people around... ;)
@nico if you want to discuss things in more details, feel free to contact me.
(1) Link works for me. (2) Is this an r coding question? How to get the Huber-White SE after fitting a model. Or something else?
Thanks mbq. One question that I still have and that I am not sure @Matt has already answered is: Do I need to pick a specific **point** on the prec-recall curve to fill in the numbers in the $F_1$ formula? (otherwise how do I compute it's value?). If so, how do I pick that **point**? (and specific precision recall numbers)?
Could you please clarify what you mean by "perturb"? Do you want to change the probabilities or the values (or both)? What is the purpose of this perturbation?
I think you have to conduct [model selection](http://en.wikipedia.org/wiki/Model_selection) to check which variables-factors can be removed. For this purpose, you have an ocean of possibilities.
Note that $\\exp(0.8747) \\approx 2.398156$.
Doesn't look like the variance is constant.
OMFG. And you are calling this sample "reasonably random"??? At the very least, these are the people who have access to the Internet (which is a skewed subset of the general population), and depending on what devices they access it on, what time of the day they need to figure out the fare, etc., you will have it skewed way further.
Thanks very much for the answer. I really like your explanation of how causality is more a consequence of design than techniques. One question I have though about techniques is whether there is something like instrumental variables for machine learning. Also -- in the case of randomization of a variable, what would the machine learning alternative be relative to a simple t-test of differences across treatments?
Would you think that one should use arbitrarily use units that prevent variance being less than one?  I would even go so far as to suggest that the units used should be such that the measure having its variance assessed should not have decimal places.  Take for example, measurements of the same length in metres and its various multiples and subdivisions.
You could also look into Stan, which might be the next big thing for Bayesian MCMC. It generated c++ code for the models; as a consequence Stan works fine with ifs and elses.
I personally find even two-way interactions between two continuous predictors very hard to understand and interpret - factor $\\times$ continuous is much easier. And ordinal (non-interval scaled) predictors should properly speaking not be in an ANOVA at all, since an ordinal predictor with values 1, 2, 3, ... will intrinsically be treated as a continuous predictor (which it shouldn't be if it is not interval scaled) - or, if we treat it as a factor, we lose the ordinality. Usually one treats such predictors (Likert scales and the like) as continuous, anyway.
(+1) Some comments: This is a good answer; it's interesting that you've taken an asymptotic approach whereas it's not clear whether the OP was interested in that or, possible, a fixed-$n$ one (or both). This answer is a little inconsistent with the OP's constraint that $U < V$, though, and if $X_1 = 0$ or $X_1 = a \\mathbf{1}$ for some $a \\in \\mathbb R$, for example, then the minimum $R^2$ for all fixed sample sizes *is* exactly $V := V(n)$. (Excuse the pathology of these examples.) Also, OLS is *not* necessarily consistent absent additional constraints on the predictors. :)
Another good book is [The Science of Conjecture: Evidence and Probability before Pascal](http://www.powells.com/biblio/1-9780801871092-1). Also [The Unfinished Game](http://www.powells.com/biblio/1-9780465018963-2)
Hi Nick, thank you for the input. I need to give this problem more thought...after finals! I would like to try out your suggestion of median/quartile errors. My advisor would like to stick to a log scale only in x since this plot is being compared to other similar plots that have been released. Thanks again.
I don't think we can really say the p-value for the t-test must be 1, since the t-statistic itself ($\\frac{d}{s_d/\\sqrt{n}}$) is actually 0/0. You can construct sequences of Gaussian (vector-) random variables representing the differences where the t-statistic is constant but the numerator and denominator converge to 0, which means that either the statistic is undefined in the limit (since it's consistent with a sequence converging to literally any finite value) *or* you're in a situation where the t-test assumptions don't hold.
I would not think in terms of "good" or "bad" as this is not a statistical category. Skewness just is.
`Why the Mahalanobis distance isn't more used?` In _most_ cases of clustering, using Mahalanobis in place of Euclidean is not much gain. Mahalanobis is Euclidean attuned to the ellipsoid shape of the data cloud. Ellipsoid or circular - the clusters _in_ the cloud can be any shape and orientation. I would be nice to use Mahalanobis _if_ one knew these characteristics (in a form of covariance matrix) for _each_ separate cluster. But you can't know it beforehand! Sooner than the clusters are discovered.
I don't think my problem lies with finding the correct model. You see the second plot in my question? The true distribution would show as a line, but due to the censoring effect it becomes a curve. I want to know how to call this phenomenon.
That would not prove any relationship, at least, not in any way that I can see.
...or you could directly fit the model $x = \\frac{1}{3} \\left(\\frac{2 y}{a}+\\frac{2^{1/3} y^2}{\\left(27 a^4 b^2-2 a^3 y^3+3 \\sqrt{3} \\sqrt{27 a^8 b^4-4 a^7 b^2 y^3}\\right)^{1/3}}+\\frac{\\left(27 a^4 b^2-2 a^3 y^3+3 \\sqrt{3} \\sqrt{27 a^8 b^4-4 a^7 b^2 y^3}\\right)^{1/3}}{2^{1/3} a^2}\\right)$ :-).
(Cont.)  But with $\\alpha=1$ double centered matrix convey values which are obtained as infinite, produced by Taylor series expansion. The key point (that I can't grab) is that somehow these values of $A$ then relate to the empirical _characteristic function_ of the distribution, and this is that relatedness which make dCov to be a measure of any, linear and nonlinear, independence.
Did you just delete your comments? I'm sorry I'm actually awful at this!
It always seems reasonable that you would want to know both, however, the optimal way to go about the process differs, so you need to make choices about what you want. The reason I asked is that most people in science are after understanding, but that goal is inconsistent with your statement "I am less concerned about the bias or efficiency of the betas". For info on the distinction b/t them see [these](http://stats.stackexchange.com/questions/18896/) [two](http://stats.stackexchange.com/questions/1194/) previous CV questions.
I would NOT transform Likert type items in any way. Items that are very highly skewed, with only 7 levels, are items where nearly all gave a response at the end. These items may not be useful at all; they add little information. 

I would first look at Cronbach alpha for items that don't correlate well with the others. I'd consider deleting the highly skewed items.

However, all this depends on SUBSTANCE and what the actual questions are. It may well be that factor analysis is not the best tool.
Do you have a specific alternative hypothesis in mind?  Note, too, that comparing the mean to the median (in some way) often shows up as a test of *skewness*: that's a good way to search the literature.  Check out the recent paper in JSE that shows up as a top hit: [Investigating the Investigative Task: Testing for Skewness / An Investigation of Different Test Statistics and their Power to Detect Skewness](http://www.amstat.org/publications/jse/v18n2/tabor.pdf)  It reports some simulation results of several tests.
What hypothesis are you testing?  That information is necessary for determining whether a one-tailed or two-tailed test should be run.  (The data cannot tell you which one would be appropriate.)
I asked this question again with more details because I still feel unresolved and do not want to open up an old question. The analysis won't be simple if the DGP is something like: $Fire = I[I(Fuel > c_1, Spark > c_2) + I(Fuel > c_3, Lightning > c_4) > 0]$, in which `I()` is an indicator function and all $c_*$ are threshold constants. Fuel is a necessary condition to Fire -- that can be detected by a simple graph. But with a lot of potential variables, we can't graph everything. I'm asking about what to do in that case.
@someben, no I don't think it makes any difference.  The issue is more fundamental than that.
Alas, questions asking for datasets are off topic here.
Obviously the question is: how to fit such a model with R or SAS ?
*Why* are you looking for a metric? After all, this search--as you point out--will not be fruitful, because your space has all of four elements! To make this an answerable question, please consider providing more details about your investigation objective.
Thanks, but that still doesn't cure the problem: the relationship you assert between the off-diagonal elements of the inverse and the co-variation does not exist.
ahh the fundamental difference had escaped me, thanks!
Meanwhile I re-read the random forests chapter of Hastie's et al "The Elements of Statistical Learning", which confirms the above statements.
What does your formula mean? It clearly is neither a probability nor one of your 30 numbers, so it is not apparent how it could be used to answer your question.
You might be interested in http://stats.stackexchange.com/questions/13984/combining-two-time-series-by-averaging-the-data-points ... it's a related problem, and has some worked examples
I even added pictures now :-)
@KMC If you want to improve your question, use edit and reserve answering for actual answers.
In SPSS, CatPCA is a part of Categories Module. Check if you have menu item Data Reduction > Optimal Scaling. If you don't have it, that means you don't have the module installed. A good and simple description and example of CatPCA usage is [here](http://web.it.nctu.edu.tw/~qtqm/qtqmpapers/2010V7N2/2010V7N2_F1.pdf).
For info, I finally implemented IRM myself (Kemp's code, without comments, is really hard for me to follow) with and without split-merge. I will eventually upload the code (python) to github, since I think it is more readable than that of Kemp. The downside is that it looks like computational efficiency could be substantially improved.
@SQLServerSteve  No, you would not substitute $x$ in there - it's not a function of $t$ any more than $\\sum_{k=1}^{k=n} k=\\frac{n(n+1)}{2}$ is a function of $k$. In fact the expression is *already* a function of $x$, and so if you had a value for $x$, you'd replace $x$ with that number.
Sorry for the delay - the change in moisture content is, I believe, just an artifact of the time. The time is irrelevant (it's just that it takes time to dry the samples out). The plot is y vs. x with y = instrument reading of moisture content vs. x = actual moisture content determined by weighing. Is that more clear?
Thanks for the response! But I think you are referring to parametric bootstrapping methodologies, the second methodology mentioned in [this](http://stats.stackexchange.com/questions/64813/two-ways-of-using-bootstrap-to-estimate-the-confidence-interval-of-coefficients) question. I wanted to learn the first one.

Still to clarify if I understood your way correctly; when you say original data, do you refer to "all sample"? Or is it just the initial random sample you took for an initial model development? From Harrell's paper, I got the understanding that it's all sample..
I don't find the wording clear here. Are you testing for normality but want your test to be robust in the face of non-normality? Or something else.
thank you for that! That was a good summary :)
For *what exactly* do you want a proof? Your question is quite imprecise, somewhat like asking for a proof that even numbers are more even than odd numbers.
Could you please indicate which R package you are using for "rank methods" and "M-estimators or MCD-estimators". The discriminant analysis function "LDA" in MASS package has only options for 3 methods ("moment" for standard estimators of the mean and variance, "mle" for MLEs, "mve" to use cov.mve, or "t" for robust estimates based on a t distribution). I know there is a method=nonparametric option in SAS "proc discriminant", is there any correspondent method in R?
Repeated measures ANOVA does not deal well with  missing data and assumes sphericity. It is not a good choice here.
here is a similar (unanswered) question:
http://stats.stackexchange.com/questions/13085/shannon-entropy-for-non-stationary-and-non-linear-signal?rq=1
How *much* more important are the negative values?
I would appreciate a discussion about why the 3 experiments would be analyzed separately, as opposed to building a grand model, and what is the best choice for the grand model.
@forecaster (Reason I say BIC, is that it places an extra penalty on complexity. In this case, the BIC of the ARIMA(1, 0, 1) has lower BIC than the model `auto.arima()` settled on, so if those were the only two candidate models, the simpler one would have been selected...)
'y' only took values 0 or 1, and the running median plot is just extrapolating a trend too far. I should have truncated the x-axis. But your comment on the tight left turn makes a lot of sense. Thanks for looking at this, and for explaining why the Stata 10 command works.
(+1) Excellent, I was looking for such an example for a long time. This shows the power of simultaneous versus consecutive testing.
I found example 10 in [Peter Kennedy's paper](http://www.stat.columbia.edu/~gelman/stuff_for_blog/oh_no_I_got_the_wrong_sign.pdf) very helpful in understanding this.
Hi @Macro . I guess some sort of robust regression, although I can't say, without research, which type would be best.
You might find [this](http://stats.stackexchange.com/questions/45043/maximum-likelihood-estimation?rq=1) answer to a similar question useful.
Could you paste a sample of your data together with the error you get?
@rolando2: I am (pretty) sure whuber meant *with* not *without*, which explains why there would still be uncertainty after taking a sample of size $N$. (Also, there are two Peter's below.) I admit the link to SRS does allow for a bit of doubt. :-)
Nothing that will inspire anybody to take up mathematics. Stick to books.
Are $\\Sigma$ and $\\Delta$ known? Further, is this self-study or homework?
Thanks for editing, but the same potential to be misleading still exists.  In the second bullet, despite all appearances, there really is only one effective parameter, equal to $a_1 \\frac{a_2}{a_4}$ (because $a_2/a_2=1$ in the argument to the exponential makes $a_2$ disappear) and it enters *linearly* into the formula, not in a nonlinear fashion. The point is that linearity is a *mathematical* property of the function and not a mere *syntactic* property of how it is written down.
ROC curves have little to do with model fit.
Could you clarify what the response variable is for one channel on one subject in one trial? Are there multiple measurements, and if so, what is the relationship between an individual measurement and a 'time segment'?
(ctd)... $\\quad$ Note that I've made some edits above which might help. In the context of my above example, with a different sample (this time of size $m$ rather than $n$, say) - the form of the ML *estimator* would still be 'the largest observation in the sample', but the *estimate* would be different (even with the same $\\theta$), and the bias would also typically be different (because of the sample size effect).
The expectation of the Cauchy is undefined. It's not $\\infty$.
You can dig the source of spectrum, and you will find that cross-spectra are computed in an object of name pgram. Or, since the coherency is defined as the cross-spectrum divided by the square root of the product of univariate spectra (i.e. f12 / sqrt(f11*f22) ), you can find what you want the two gains as tou define them by multiplying the squared coherency by either univariate spectra.
What are you asking? You haven't provided any of the data needed to conduct any tests at all.  Is this perhaps taken from a multiple choice question that is probing for an *appropriate* or *maximum* number of intervals?
(1)Don't responses come from different individuals independent in fixed effect model ? If so then why the assumption is $\\sum_{i=1}^{a}\\tau_i=0$. Doesn't it imply $\\tau_i$ are dependent?
@ayush biyani: The behavior you are experiencing: good accuracy on one dataset, with poor accuracy on another, is usually taken as an indication of overfitting. However, it could also be the case that some sort of "structural shift" has happened between the 2 time periods, and you need a different model for the old data. Your concept of "shock proof" is probably analogous to "overfitting:" in both cases removing certain variables allows you to generalize further from your model.
Sorry about any ambiguity: it's the never-ending debate out there that is wearisome, not your question.
Are you talking about estimating a model with $k$ parameters, but only using $m<k$ to predict, without changing the estimated model? Or are you talking about reducing the number of parameters in the estimation?
This sounds quite similar to biological sequence alignment, which has been studied quite a bit. You can find some information here: http://www.ncbi.nlm.nih.gov/BLAST/tutorial/
(1) You still need to *fit* the models by maximum likelihood, if they're not completely specified, & (2) you want to penalize the likelihood for the model that has more free parameters (see AIC), if one does.
It's unclear which "ILP" you are using.  Otherwise, you are asking largely for opinions about what you should do and about what is good or bad.  This site is better suited to questions about how things work or about what the truth of a matter is.
If the model is linear in its parameters, the average prediction should be the same of the prediction of the average of the parameters. For other aggregation functions and/or nonlinear models, the results will be different. Personally, I'd be wary about messing around with the parameters in that case.
Every case I've ever seen has poorer fit when you use all items than when you use fewer items or you use item parcelling. However, it is **wrong** to conclude that you should therefore not use individual items. Researchers should not blindly apply the same rules of thumb (e.g., rolling out the Hu and Bentler citation) for reasonable fit to models of very different complexity.
This is a very good text but is not concise and emphasizes design rather than provide a general introduction to statistics.  I think that many of the responses are getting away from the original question from the OP.
Absolutely. It can be interpreted as a reduction of V1's inverse estimated effect across levels of V2 (or vice versa), i.e. the inverse effect of V1 isn't as inverse for higher observations of V2. You should plot everything to verify.
I will try checking out the hat matrix now. I tried an example with fairly uncorrelated predictors ($\\rho_{ij}=0.25^{|i-j|}$) and got the following: 
The data: http://ohyur.com/random/lowdata.png

The estimates: http://ohyur.com/random/lowestimates.png .
So even while the pairwise correlations in the other data are much stronger than these, it is still only the adjacent predictors that are correlated.
That's what I was looking for. Thanks a lot.
Can you write a Python script that will recode your `speaker` variable into a set of new variables `speaker1`, `speaker2`, (etc), with some instances where more than one variable has a 1 in it? Would that solve your problem?
*Please* don't refer to *Wikipedia* as '[wiki](http://en.wikipedia.org/wiki/Wiki)'. That's a bit like talking about *using a program someone wrote in Java* as 'using Java', or calling a book written in Word 'Word'. Wikipedia is an example of a wiki, and is implemented in wiki software. *Crossvalidated* is arguably a wiki.
Thanks! But I hoped for a statistic that is faster and more stable to calculate...
This sounds like you are asking for a computer program to implement the criteria stipulated in the question.  If so, we need to migrate this question to SO.  Otherwise, you need to clarify what you mean by "determine the trend."
We won't be able to answer your question if you don't tell us what $N_k(x)$ is.
@whuber: maybe the $x$ and $y$ are stand in for [$p(x)$ and $q(x)$](http://phdfb1.free.fr/phdthesis/node70.html)
But what's about the last question: "What are the other methods/algorithms that I can try?". Actually the answer that I got concerns implementation of the methods which is a topic of CV, or am I wrong?
This is more a comment than a complete answer.
I am surprised that none of the answers here mention that usually it is meant in the mathematical "linear algebra" sense of the word.  For example, when we speak of an "orthogonal set of variables" usually it is meant that $X^{T}X=I$ for the matrix with the set of variables $X$.  "orthonormal" is used as well.
So what is the problem, then? Lack of homogeneity in the variance?
Added two relevant links in my post below.
Also do you have access to the underlying data? How are the curves drawn?