tttjjj/elt_bench huggingface dataset:
@MichaelChernick, I don't think there is "so much" discussion about how one writes things. The guidelines we've tried to help you with are more along the lines of "if everyone did that, this site would be very disorganized and hard to follow". It may seem like there is a lot of discussion about these things, but that's probably just because you've been a very active participant since you joined, which is great, since you clearly bring a lot to the table. If you want to talk more about this, consider starting a thread on meta rather than a comment discussion under my unrelated answer :)
Thanks a lot, Tom! I`m pretty busy right now and also have a idea how it could work in R. But I`ll definitely have a look at your proposal asap!
Hi @Gideon, no probs, if it was useful could you consider clicking on the "tick" mark next to the answer to accept it, so the question is removed from the "unanswered questions" category.
+1, been there, done that. Explaining why R does things in certain ways compared to other software can add significant time and burden to the class that you may not otherwise account for(i.e. reference levels for factors, etc)...or just simply getting data into the right format can be a challenge.
You say that there are two populations.  Are these two pre-existing populations from which you've drawn random samples or are these two hypothetical populations to which you've applied some treatment through random sampling?
I'm simply trying to show that the sample mean is an unbiased estimate of the population mean.

$x_i's$ represent samples and $X_j's$ represent population values.
B Seven, your question is much too high level and not very specific. Basically the whole field of machine learning can be boiled down to the question of how to avoid overfitting. There are several strategies like cross-validation, regularization or using a proper prior. Every good machine learning book can help you with that (e.g. the Duda/Hart/Stork or the one by Bishop). It is also not clear what you mean by a "tainted validation set". If your model cannot cope with changing time series data, it means that it is probably too simple. But more complex models will need even more regularization.
Thanks stat. I will try to look into that.
Thanks for the answer. This 'class imbalance' problem seemed to me to be most obvious with Naive Bayes since the Bayes' Theorem equation contains a specific term for the class prior. From what you wrote about class imbalance being a general problem, am I correct in assuming that other ML classifiers like C4.5 Decision Trees and SVM also implicitly or explicitly use class prior probabilities?
@ayush When you say "dependents", do you mean like "this person has five dependent children".  If so, what happens if someone has 0 or 5 dependents?
BobL, I've added a few links to some other, somewhat related questions where you may find the discussion helpful
I don't use your software, but I seem to have predicted one of your difficulties added in later editing: as said, probability densities can exceed 1 _and this is in no sense a problem_. Think of a uniform on [0,0.5]: the density must be 2 for the density to integrate to 1 over the whole distribution. But you just need to consult the documentation for your software. If this isn't explained, you need better software.
If it's a matter of generalising from experience I would say that positively skewed distributions are the most common type, especially for response variables which are of central interest, and that lognormals are more common than normals. A 1962 volume _The scientist speculates_ edited by the famous statistician I.J. Good included an anonymous piece "Bloggins's working rules", containing the assertion "The log normal distribution is more normal than the normal". (Several of the other rules are strongly statistical.)
"I want to know what choice people usually make, and why." Choice as far as what? I think a considerable amount of detail is missing from this problem description. Concerning sampling, if you state an actual scientific question, there's no need to sample the whole network, just a subset of nodes to produce conditional independence among relevant nodes. However, you can make no statistical inference by only sampling a focal node, you also must sample other similar individuals and their nodes for conditional independence to make inference on variations in path structure/weight.
@JAW I'm interested in the correlation because one of the predictors I'm using in my actual analysis is novel, and I want to demonstrate that it's not too related to other predictors.  I suppose you're right that it's not really a problem of survival analysis, but I want to make sure that the survival structure of the data doesn't cause problems for my correlation calculation.
`screen` or `tmux` are better than `nohup` -- they also detach the script so it won't be killed by logout, but allow to reattach the session and start over, even from other client computer. `tmux` can be even used as a kind of text-mode window manager.
A kernel density estimate would be a nonparametric *estimate* of a distribution.
Your coin doesn't know anything about the previous flips.  It's a 50% chance regardless of what has already happened.
Thanks for the intricate response. The new tool provides more data and costs less, yes. I ended up doing a lot of what you describe including a correlogramme of the values after matching the data points.
I don't know anything about HMM and I'm sure this is a fine answer (+1, btw), but this seems a bit extreme for a "beginner question"!
"Makes sense?" is a question, not an answer. If you're asking a question (like "Here's what I did, does it make sense?") it belongs in an edit to your question or in a new question of its own. If you're actually answering your own question ("Here's a solution to the question I asked, and here's why it makes sense"), then it's an answer. Pick one.
+1. If you're unfamiliar with $\\TeX$, a good way to use it anyway is to right-click on relevant expressions in the question, choose "Show Source," then copy and paste them into your answer.  You can then make modifications, which usually are intuitive and obvious.  This will make your replies more readable.
Thanks a lot @Arne, this helps. I found the article by Verbeek and Nijman most helpful. Sorry for not stating my research question explicitly. The goal here is to compare the two kinds of data much like what is covered in these articles, we will be applying it on a new kind of data-set, where the comparison might be better.
What if you combine categories 1 and 2? This is commonly done in ordinal scales where one category has very few responses.
Try `?mlogit.data()`
(+1) Well said!
Side point: the notation is for *independence* which is not necessarily the same thing as *orthogonality*, although independence and uncorrelatedness / lack of covariance will coincide in the Gaussian case.  (See also @whuber's comment on http://stats.stackexchange.com/questions/12128/what-does-orthogonal-mean-in-the-context-of-statistics )
Could you please describe the situation in more detail? Are the groups equally large? How large? Are there drop-outs? How do you want to deal with unequal group means at baseline?
Just as I expected here: http://stats.stackexchange.com/questions/15805/mean-and-variance-of-a-normally-distributed-random-number-created-from-the-averag. This is a kind of transformation that is covered in any calculus-based introduction to statistics book, and you have half of the answer in your question already: $E[Y]=a\\sum E[X_i] + b$, $V[Y] = a^2 \\sum V[X_i]$.
Bowerman O'Connell and Koehler is quite good for introducing ARIMA models but I don't think it includes ARIMA with covariates. You could try my 1998 textbook which covers ARIMA modelling and regression with ARIMA errors at an introductory level. See http://robjhyndman.com/forecasting/ for details.
Yes, that's an intuitive way to justify why in this case we need to scale (multiply) by $T^{1/2}$, so that $\\bar X_T$ retains a distribution. A smaller scaling would still result in the variance becoming zero, a larger scaling would result in the variance becoming infinite.
Ok I got the answer from elsewhere.
In this context normed means the integral over the histogram is equal to 1.
@AJMansfield This is a bit like saying "every distribution is discrete" - while literally true, it obscures the relevant issue. A typical number of bins in a binned estimator is vastly smaller than a typical number of pixels... and with any graphics that make use of anti-aliasing, the 'effective' number of pixels is larger (in that it's potentially possible to distinguish differences of positions between pixels)
@AdheshJosh: I think you have to click some symbol (I don't remember what) to award the bounty to MånsT.
Well, I guess that a some point in your clustering procedure you introduced a distance between points, between clusters, why not use these ?
See http://otexts.com/fpp/8/9/
For 1. -- in what way? The residuals are effectively already in the sample likelihood.
Could you indicate *why* you are fitting distributions to these data and what you hope to accomplish with the result?
That seems to me like a related but more theoretical problem (and, admittedly, I have not fully understood how to read your measurement results). My suggestion was completely pragmatic: if your program runs for milliseconds, let it run many times and compare the averages. This is what all the fancy statistical tests would do if they could ... (but if you are interested in these tests, maybe you should edit your question and pose the problem in your comment)
"BUT i need a value to check if this difference is above or not...if i dont have the percentage how can i compare them?"  What value you set as the threshold ($30\\%$ or $10\\%$ or whatever) is not something that you determine from the data.  This value is determined by _you_, or by your professor, or by the client "I will accept variations of up to $25\\%$", and then you have to find which of the data satisfy this externally imposed criterion and which don't.  The data $A=10,B=30$ don't tell you that $25\\%$ is the acceptable variation.
Thank you for posting this.  It's an important (and potentially inspiring) discussion to have.
http://en.wikipedia.org/wiki/Condition_number
I really appreciate the advice but unfortunately I don't have the time (or expertise) to change over :) I want to show the effect of all predictor variables in isolation included in the analysis based on their AIC and p values when evaluated in isolation. I was thinking I would report the p values from the isolated models and how this affects my hypotheses. I would also include those models with multiple predictor variables which have the lowest AIC values overall and discuss these models, the isolated models and an all predictor variable model. I hope that makes sense :)
I arrived at a very similar result with a Matlab simulation, but I was curious about how I would solve this analytically. Also, since I play with my kids, they all want to finish the game, regardless of who wins, so I do want to ask about the maximum.
This is a good suggestion, but given the OP's evident level of statistical sophistication is likely to be too advanced.
In the fully nonparametric case, the uniformly minimum variance unbiased estimator is $\\frac{2}{n(n-1)} \\sum_{i < j} X_i X_j$. It's not guaranteed to be positive, though.
I wonder how well these bounds compare to the elementary Normal approximation based on computing the means and variances of the truncated distributions.
Your question lacks some critical details -- first of all, what should be the protocol of this forecast, i.e. you want to predict the ozone level for next day every day? Predict the mean level in 2020 from 2004-2006? Next, is the ozone levels the only data you have? How it is discretized, one measurement per week/day/hour? Maybe you can pull some other relevant data from other sources? What can be relevant from chemical point of view?
If you have added several Likert items into a single scale, *you have already assumed an interval scale*. The only remaining question is whether you particularly seek linear correlation or more general monotonic association
Short answer, yes. You can prove it directly using $O_P(1)$ definition.
The changes of support is still a concern, but a proposal is to do daily interpolation from the monthly (and weekly) data. Possible interpolation schemes include 'last observation carried forward' or linear interpolation.
(+1) This is a good answer. I think there may be a way to further draw out a very important point, though. In the way the bootstrap is normally carried out, there are *two* effects that are happening. First, we are pretending that the sample we have obtained is a proxy for our population. This is nominally a reasonable thing to do, provided our sample size is reasonably large. However, we usually have a hard time calculating the actual quantities of interest from that pretend distribution. *So*, we have to *estimate* them, and this is why we draw lots of bootstrap samples. If we could.../...
What are the counts? Are they all small numbers or do they vary over a wide range?
I can write JMP scripting code or SQL or something else to calculate the values if I have the formulas involved.
Welcome to our site. We expect textbook questions like this to exhibit some amount of research or effort towards their solution (which can readily be found in other threads). For more information please see the [help].
Are your variables really 0-1 _dummy_ or simply 0-1 _binary_?
Generally, contrapositive is indeed only absolute!

There could be a condition that is needed for this contrapositive of probability to work.  I don't know if this is anything formalized or if there are similar ideas.
Why don't you edit (change) your question to that, @Cynderella? I think that's a viable, on-topic, non-duplicate question.
@whuber. Actually, I used a software to make the fit http://www.mathworks.com/matlabcentral/fileexchange/25948-variogramfit. It allows me to specify whether or not to add nugget when I fit a model variogram. I fits the variogram by minimizing the least squares error. If I don't fit the nugget it will be zero. But if I allow it to have the nuggets, it comes like this. Actually, I specified a gaussian markov random field by its precision matrix so that every node is connected to its 3x3 neighborhood. From this precision matrix, I got the $\\Sigma$ matrix and then sampled from the gaussian.
@NickCox I agree M1 is linear, but M2 is not. The fact that I have to compare a linear (M1) to a non-linear model (M2) is the reason for the question. I am using nls for M1 just to be consistent with M2.
Is treatment randomly allocated? Is allocation concealed? Are the participants blinded to the treatment allocation? How long do you expect it will take for the treatment to have an effect?
Thx.. relsurv seems to be what I´m looking for. But the methodology seems to be very similar to what I sketched above,except for the cox analysis in relsurv instead of kaplan meier analysis.
"It's not unlike folding the distribution over at the average and taking the average of that." That comment, like the rest of your post, seems to describe mean absolute deviation, not standard deviation.
The answers you are getting seem to be missing the question you are really asking and are hung up on least squares analysis, which is not what you want. Searching our site for [errors-in-variables regression](http://stats.stackexchange.com/search?q=errors-in-variables+regression) and [PCA](http://stats.stackexchange.com/questions/tagged/pca?sort=votes) will head you in the right direction. In the meantime, think some more about what you really mean by "close as possible": it's easy to construct (infinitely many) curves that pass *exactly* through your points, in sequence.
Indeed, the analogy with the $t$ is quite direct, since the Mann-Whitney statistic is monotonic function of a t-statistic computed on the ranks. The precision of a difference in population means in the t-test is a simple function of $n_1$ and $n_2$ (*ceteris paribus*), and the same thing applies to the Mann-Whitney in respect of the difference in expected mean rank for the two samples. The same apt analogy with area that you mention in your linked answer applies here in the sense of maximizing the precision of that difference.
Seems to me that depends on the largeness of bins.
If you're able to calculate the likelihood of $x_1 = 3$, $x_2=6$, $x_3=2$, *you're using the wrong likelihood*.
These are measurements of two parameters of a motor. Both the variables are related. But when i check the correlation coe-fficient, they are negetively related.
That's a neat proof. I don't suppose that theorem has a name by any chance?
Explicitely, you’re computing $$ \\int_0^{1.96} f(x) \\phi(x) \\mathrm d x$$ where $\\phi(x)$ is the density of the uniform distribution on $[0, 1.96]$; so you compute
$$ {1\\over 1.96}  \\int_0^{1.96} f(x) \\mathrm d x.$$
Hi Dave.  I never heard the term intermittent time series before.  Does that mean that it is 0 for a time then behaves like a statioanry time series then go back to 0 for a while and so on.
@gung This is not homework, it is for my own edification. I saw the problem in David Mackay's textbook on information theory and was not satisfied with his explanation (or other similar explanations on the net).
(-1) To this response; this is clearly a theoretical question ;-). Consider improving why you think I should code it in BUGs or STAN and what it does have to do with the original question?
May not be any help to you, but note that sorting the data is *not* the most efficient way to compute percentiles. Sorting takes O(n log n) operations, while [selection algorithms](http://en.wikipedia.org/wiki/Selection_algorithm) exist that take O(n) operations.
Out of curiosity, is the misspelling of "theoretical" in the plot built into the matplotlib and scipy.stats packages?
@AlecosPapadopoulos The OP wrote "the density is some constant **over** $x$," which I interpreted to mean $f_X(x) = c/x$ for some constant $c$.  This is correct.
Looks like TN vs FP is high, but TP vs FP,FN is low. Your problem lies in TP. It means your classifier is mostly right when a sample test is negative, but unreliable in the opposite case. This is plausible (though not satisfying).
Because the noise model is not conditioned on the input.
My answer is wrong, as I was thinking of it as the independent variable not the dependent variable.  I guess I should delete it?  Let's just say I'm answering a different question.
I do not understand the distinction you are making at the end between "capable of compilation," which is a matter of implementation, and "procedural," which describes the language. Incidentally, Fortran and C are both procedural and (usually, but not always) compiled.  Did you perhaps mean to write "interpreted" instead of "procedural"?
Thank you very much for the simple but useful reference.
Thanks, that's what I wanted to know: rescaling to the grand mean (ignoring the group structure ...).
Are the three quantities you call X, Y and X measured in the same units? If not, simultaneous fitting would require somehow normalizing those values. Minimizing the sum (for the three variables) of the sum (over all points) of squares only makes sense when the three variable are in the same units, or normalized to be so.
There are only two steps for fixed $(\\mu,\\Sigma,\\epsilon,\\delta)$: (1)Simulate from a MVNormal$(\\mu,\\Sigma)$, (2) transform each entry using $\\sinh[\\delta^{-1}\\mbox{arcsinh}(x_j)+\\epsilon$ *et voilà * and now you have an answer.
1. p(b|c)/p(b) = p(c|b)/P(c) not p(b|c) = p(c|b)/P(c) right? 2.page 374  formula after (8.26)
Or this one http://stats.stackexchange.com/questions/30/
A little research on our site ought to turn up explicit answers.  Keywords to look for include SVD and PCA.
Your linked article has the points you make, in section 3 "Include All Constitutive Terms."
I dont think this answers the question! It answers __another__ question!
I am working on address field of a database. My strings have varying length; say 4 to 6 words. Please can you break down how to generalise the hamming distance to allow for insertion?
Formulas are in many places, such as [Wikipedia](http://en.wikipedia.org/wiki/Student%27s_t-distribution). The problem is that some of them may require numeric integration or approximation to compute; your ability to do that depends on the application platform. The answer is trivial if you're using `R` or even Excel but is a little involved if you're writing in Assembly language! You need to tell us more about your application and its constraints.
Is this a homework problem? In any event, the statement that errors are uncorrelated is an *assumption* of the AR(1) model, not a (genuine) derivable result.
For the record, because it's not clear on that Web site, this .dta file is in Stata format.  It has no apparent problems.
I would probably add, that the difference between regions in terms of occaisonal smokers is largely restricted to the second axis of variation and as we can see in the stats this is such as small component as to not really be worth interpreting. The structure in the data is really the contrast between BC and the other regions in terms of daily smokers and former smokers.
@naught101 - At this point I can't think of anything :-)
+1. For `has some localization feature, namely that larger "distances" should not be preserved but local distances should`, however, one could note that MDS - good implementations - provide option to weight input distances. So, one can put more weight (importance) for small ones. But, in any case, a good distance function must be crucial. I was somewhat upset that t-SNE (which's new to me) `led to no discernable different results`.
Could you please explain what you mean by "vector" and how you calculate a z-score for a *single* vector? You don't seem to be using the term "vector" in its standard mathematical sense. This is an unusual context for z-scores and it admits of several natural but different interpretations of "z-score."
(+1) This question may seem confusing because what the OP has been told is *exactly wrong*--but that's what makes it a great question!  It invites us to explain how to interpret the statistics and plots shown here.
Sure--that's a routine procedure to maximize the PDF given its functional form, as taught in Calculus. What @Hong Ooi is alluding to, though, is that since your question mentions *data*, then those parameters apparently were estimated from the data and therefore are uncertain. The questions of statistical interest become (1) how does that uncertainty translate into uncertainty about the location of the mode and (2) why are you estimating a mode in the first place? The latter is important because it determines *how* the parameters should be estimated, which in turn influences the answer to (1).
<p>I am trying to read about graphical models from http://ai.stanford.edu/~paskin/gm-short-course/lec2.pdf. </p>
Here are the equations I am confused about:
<p>$\\sum_{e}\\sum_{a}\\sum_{m}p_E(e) · p_B(b) · p_{A|EB}(a, e, b) · p_{J|A}(true, a) · p_{M|A}(m,a)$
<p>=$\\sum_{e}\\sum_{a}p_E(e)·p_B(b)·p_{A|EB}(a, e, b) · p_{J|A}(true, a)·\\sum_mp_{M|A}(m,a)$ </p> 
<p>=$\\sum_{e}\\sum_{a}p_E(e)·p_B(b)·p_{A|EB}(a, e, b) · p_{J|A}(true, a) . \\psi_{A}(a)$.</p> 
<p>I am assuming $\\psi_{A}(a)$ is pmf of A. Ideally, $\\sum_mp_{M|A}(m,a)$ should be equal to 1. What am I missing? </p>
Is there censoring (dropout, other disease)? Do you assume that it is the same for both trials?
@MichaelChernick is Linear Discriminat Analysis a valid solution to my problem? I have both categorical data and categorical classes
@whuber All you have done there is collapse all the infinite >=3 children scenarios into the 2 boys catorgary. So you get 0.5:(1G:0B) 0.25:(1G:1B) 0.25(0G:2B). Which gives a total of (0.75G:0.75B)