tttjjj/elt_bench huggingface dataset:
> I also know that $X$ and $Y$ are independent...   You mean that $P\\{X=4, Y=13\\}$, the probability that the hand contains all $4$ aces and also has all $13$ spades, equals $P\\{X=4\\}P\\{Y=13\\}$, the probability of $4$ aces times the probability of all $13$ spades? Since $P\\{X=4, Y=13\\}= 0 = P\\{X=4\\}P\\{Y=13\\}$, which of $P\\{X=4\\}$ and $P\\{Y=13\\}$ would you say is $0$? In any case, if you _know_ that $X$ and $Y$ are independent, there is no need to check covariance: independent random variables are uncorrelated random variables (subject to existence of various expectations, of course).
@onestop The pulley central to Atwood's machine has markings at regular intervals. A laser sensor is directed at the pulley; it measures how quickly the markings pass. The information is processed by a computer, which just gives us acceleration directly.
your last explanation on finding the smallest intersection can be applied to all cases. Min `A∩B` is `max(a+b-1,0)`
Thanks for pointing out Glen. Was careless there looking at another similar question. Double checked, the workings worked out to be right. =)
I have implemented absolute error loss in my MLP library, which uses scaled conjugate gradient descent.  IIRC the lack of a derivative at zero doesn't actually cause much of a problem in practice as you know where the discontinuity is, and the error of the neural net is very rarely *exactly* zero.  I'd have to check my code when I get back to the office, but I don't think any special code is necessary.
The issue of underlying parametric assumptions motivates a discussion about fully parametric vs. semi-parametric or *non*-parametric inference. That is not a ML vs Bayesian issue and you're not the first to make that mistake. ML is a fully parametric approach, it allows you to estimate some things which SP or NP can't (and often more efficiently when they can). Correctly specifying the probability model in ML is exactly like choosing the correct prior and all the robustness properties (and sensitivity issues) that implies.
The main reason why I ask is because I am using MATLAB for my algorithm and I couldn't find any Jenks natural breaks optimizations in any toolboxes etc. so i will need to implement my own.  I just wanted to know how much better/faster this might be before I switch gears and implement this.
Irrespective of who (if anyone) thinks so, no, the second case is not a subset of the first.  The second case explicitly excludes interest in parameters, which are the focus of the first.
One pitfall is that such a regression often makes no substantive sense.
Thanks for the suggestion. I might actually try that first just to see the results. To clarify, do I just generate random values according to the distribution of $\\overline{X} \\sim N\\left(\\mu, \\frac{\\sigma^2}{n}\\right)$ and $n\\frac{S^2}{\\sigma^2} \\sim \\chi^2(n-1)$, then substitute back into the expression to calculate the corresponding value for $D$, then apply a kernel density estimation technique to estimate the distribution for $D$?
@ClaytonStanley Since the small-sample distribution of the variance depends heavily on the distributional form of the data, relying on a distributional assumption may not be worth the computational "saving". (Or to put it another way - "*if you don't care if it's right, you can have it as fast as you like*"). Indeed, since you don't even state a distributional assumption, on what other basis than this sort of approach could the question even be answered? It would require guessing what your assumptions are.
Can you tell us more about the data? Is it plausible that the effects are legitimately that large? If not-- in some situations (e.g. some kinds of case-control data) non-independence GEEs give inconsistent parameter estimates. Also, is it possible that complete/semi-complete separation (e.g. see the answers to this question http://stats.stackexchange.com/questions/27442/logistic-regression-performance-with-high-number-of-predictors) is an issue? This produces unstable regression coefficient estimates
I am not lecturing: in my role as moderator I am pointing out points of potential confusion and ambiguity in your question and requesting clarification, without which you might not get helpful answers. Since you claim to have a hypothesis, please edit your question to indicate its nature and how it was derived (because the answers, as of course you already know, depend on whether you developed it from the data or independently of the data).
Actually, it's probably *not* a good idea to think of $f(x)$ as $\\Pr(X=x)$ because, when a density exists, $\\Pr(X=x)=0$ for all $x$.  This way of thinking makes unnecessary mysteries of some common phenomena, such as the [existence of values](http://stats.stackexchange.com/q/4220) for which $f(x) \\ge 1$, an impossibility for any actual probability.  For a better alternative, please see http://stats.stackexchange.com/q/14483.
$E(X^4)$ is not uniquely determined by the mean and variance. An ${\\rm exponential}(1)$ random variable has mean and variance equal to 1, but has fourth moment equal to $4!=24$ while a normally distributed variable with mean $\\mu=1$ and variance $\\sigma^2 = 1$ has fourth moment equal to $10$.
Binning is not the way to achieve your goal.  It makes the relationship discontinuous will can ruin the process.  If by MDR you mean the technique used in genomics, that is not a proper data reduction method, in the sense that it peeks at $Y$.  Look at things like variable clustering, principal components, redundancy analysis.
You can use a KS test for a completely specified discrete distribution if you simulate the distribution under the null. The bigger problem tends to be that the KS test is unsuitable when you estimate parameters, and that problem holds for the continuous case as well. e.g. you can't test 'normality' by a KS without estimating the parameters, in which case the tables don't apply. If you estimate parameters you have to simulate the distribution of the test statistic, leaving you with a Lilliefors' test.
Welcome to the site. Usually answers, as opposed to comments, provide more fully worked out solutions, but I think this qualifies as an answer
Possible duplicate [here](http://stats.stackexchange.com/questions/61131/test-cox-proportional-hazard-assumption-bad-schoenfeld-residuals), possible partial duplicates [here](http://stats.stackexchange.com/questions/82837/goodness-of-fit-testing-cox-proportional-hazard-assumption-in-r) and [here](http://stats.stackexchange.com/questions/15329/how-should-i-deal-with-the-consequences-of-proportional-hazards-violations-in-lo)
Small sample size? When I use a sample size of `2000` the type 1 error rate is `.047` - which is pretty close I would say for 1000 simulations. It appears to me that those coefficients for `glm` are Wald tests - which will only asymptotically give the correct type 1 errors.
@PeterEllis: Both parts face the same question, but may have different perspectives.
It would seem to me that length of stay would require a different model than frequency of hospitalization, but that doesn't mean it couldn't be on the rhs of the regression.  Why put it in as an offset, vs. as a standard rhs variable?  Not sure, from a predictive point of view, that it's fair to include it at all, so depending on your objectives you might want to leave it out after all.
@Behacad - assuming you can defend to your potential critics the representativeness of your sample, I would say unequivocally that trying to estimate relationships among 12 variables is asking too much of your 42 cases.  See if you can simplify your model to include just the 3 most interesting predictors.  Although I realize it's painful to part with data you may have worked hard to collect!
I have been interpreting $\\mathcal{D}$ being distributed Dirichlet, but you are right! $\\mathcal{D}$ is actually the Dirichlet distribution itself! Makes so much more sense now. (I must change my question so I am not saying $\\mathcal{D}\\sim Dir$.)
Update: I broke the analysis down into very small pieces. When I have only one repeated factor (eg. time instead of time and task) the analysis comes out the same for ANCOVA and mixed model. It must have something to do with how mixed model treats the data when two repeated measures are present.
How is it possible for a time series to be both "stationary" and to have a changing autocorrelation?
Could you please clarify and/or support this statement? For instance, which stepwise algorithm do you know that applies such adjustment?
This is not really a specific programming problem. This is a statistical modeling problem. It wounds like you ware interested in some form of imputation for dealing with missing data. You should familiarize with yourself with missing data methods before trying to program a solution. It sounds you don't know how to do it anywhere and it's not R specific.
+1 - good and detailed explanation
@NickCox, you are right, the data is usually 0 or -1.  In my data, sets can contain discrete values, [-2,2].  The Y-axis is the average of the sets so it can be any real number within that range.  Most of the time the sets contain 1 value so the average is discrete.  Do you have any advised for analyzing this type of data.
Already found a good question about it in this forum: http://stats.stackexchange.com/questions/20514/books-for-self-studying-time-series-analysis.
Awesome! Thanks. Also, I believe the summation limit on $\\bar{X}$ should be 1 to g, but I can't make 1 letter edits.
Thank you. About you question why some people didn't answer or why they were not asked. For example consider the response variable 1 - have children, 0 - haven't children. The predictor "My job is secure" is only answered to employed people.
+1 This seems appropriate *in conjunction with using the data correctly.*  Specifically, each *account* should be tracked, not accounts per employee (which is a biased approach).  The times to resolve accounts will still be censored: there will always be some open accounts (or accounts closed solely because they were deemed unresolvable).  Use censored methods to estimate the mean resolution time, multiply by the number of accounts to get total time required, and divide that by the number of employees to obtain mean "length of time to resolve an account per employee."
I think I understand why. Thanks.
Can you post your crosstabs?
Ah...Merci :) that is superb.... Thankyou very much for your time and effort...this will help a lot!!! I acccepted the answer
Please fix the question, rather than giving an answer that isn't an answer. Role reversal is a good idea here: in reading other threads, you want to read first the question, then the answers. You don't want to find that the question posted wasn't even what the poster really wants to know.
Ah, I see. I am not aware of papers specifically concerned with that approach, only work that has been done with L1 and/or L2 regularization in PLS or CCA.
I will try the approach in the paper and see how it works. I don't think decision trees will work given the classes are not ordinal and one hot encoding is not feasible, I could try using them once I  reduce the dimentionality.
You are missing 4) add the desired mean of the distribution (provided that in part 2 'width 1' = standard deviation 1 and mean is 0). If this does not fix the issue, can you add the results (e.g., some plot) and information about why they are not consistent with what you expect? The method you describe, if correctly implemented, samples from the multivariate Gaussian, but maybe you want something else than samples from the distribution.
Articles directly about this would be hard to find... perhaps look up least significant difference scores... which is equivalent to what you ware trying to do.
What appears at the channel output is determined by the distribution of the input and the channel transition probabilities. However, the channel capacity is determined by the channel transition probabilities alone. Are you sure you are not being asked what rate of transmission is achievable given that the input is constrained to be such that the channel output distribution is as given?
Gosh you're right, sorry I did not fully understand the question you posed. Suppose we boil your problem down to the simplest case where all $w_i$ are Bernoulli RV. Then you are essentially observing the sum of a random subset of $n$ RVs. My guess is there is not a lot of information here to estimate with. So what did you end up doing for your original problem?
Are you supposed to consider the genotypes as nominal categories (i.e. just genotype 1, 2, or 3)? In that case, the sounds like a classic $\\chi^2$ test situation (I think this one is usually called the 'test of homogeneity' or 'test of independence').
@AndresRiofrio In response to your request, I have added one of the tag we generally use when dealing with independent events in probability.
Actually, covariance matrices don't have to be positive definite, just positive semidefinite.  If the matrix isn't of full rank, that implies the variables span a subspace of dimension less than the number of variables.  In other words, you can write the variables as linear functions of a smaller number of variables... I hope this helped w/o giving it away totally.
Hi allie, that is what @Peter Ellis mentioned in the second paragraph of his answer. His second sentence there covers the problem that this technique introduces. Do you have a theory, which is telling you what predictors to put into your model?
Be sure to read the longer explanation below!
Add the `-h 0` flag to your options string. Basically it means that no shrinking heuristics will be used by the solver.
Do we actually have any reason on the basis of the wording of the question to think that people choose their passwords randomly? If n=4 and half the people choose "1234" (... "*the kind of password an idiot would have on his luggage*") the chance of getting the password with someone else's card is quite high.
Thank you. This was helpful. Accepted as the answer.
My question was about the adaptation of DTW as a method to the context of irregular time series. Packages like Zoo do not provide a solution to that problem.
Hi, Thanks a lot for kindness answers. It is useful.
Varty, I very much appreciated your response. It was very helpful, and very quick! Not sure i'd feel comfortable explaining it just yet but I'll keep reading. If you know any readable texts on Tobit please feel free to forward them. Many thanks again
I feel that my adviser was not correct on making such a statement! I wanted to confirm this and this was the only reason for me to post the question.
Glad to know this works for you.
One of the problems with using combinations of kernels is that it means you have even more kernel parameters to tune, which means you are even more likely to run into problems with over-fitting the model selection criterion.  In my experience, using a combination of kernels will only improve performance if it is based on expert knowledge, and is likely to make performance worse if the combination is selected in a purely data driven way.
On the face of it, this question ought to have a routine answer in the SAS manuals and help files. Is there something in those references that you need help with?
@Tie-figher I tried to come up with a clearer title given that the data isn't "missing" in the statistical sense. feel free to modify.
The multivariate hypergometric distribution ( http://en.wikipedia.org/wiki/Hypergeometric_distribution#Multivariate_hypergeometric_distribution) is what you're looking for
The $n_i$s are between 1 and 200. I am interested in $X = (X_1,X_2,...,X_k)$, where $k$ is around 5 to 20. The $p_i$ for my problem are not necessarily small.
It's not clear to me how those standard errors were obtained (but I didn't read the whole paper). I came up with one or two possibilities, but I was able to show they couldn't explain the rest of the same row. Hopefully someone else can do better.
50 points is a small sample to use: you cannot expect it to estimate the variogram reliably.  What results do you get with, say, 300 sample points?  What happens when you repeat your simulation?
Can you say what you mean by "improves results" and "results become worse"?
Hey, just to get some feedback: did you look into the links I provided together with the answer? Was this helpful to you?
Can I pair the data with the Kruskal Wallis test? I.e. pair (well I suppose group is better than pair for 3 samples) the 0.4, 0.48 and 0.47
@JasMax Right, but what is the mathematical form of that model? For example, the simplest relationship could be $C_i$ is linearly related to the photosynthesis rate, so your model would be $A=m C_i +b$. However, the data clearly show non-linearity, so obviously a line isn't sufficient. Also, a line isn't a mechanism. In the plot shown you have some interesting features: a strong positive effect of $C_i$ on photosynthesis followed by saturation and actual drop off. What's the explanation for that? Can you write down a physical model of what you think is happening?
This reminds me of the "classical" survival analysis example of the heart transplant data: http://bit.ly/UFX71v - what you need is a time varying **covariate**, not necessarily a time-varying **coefficient**. You can plot your data using KM curves.
Please explain how the concept of likelihood would apply to your "non-stochastic" model.  Where exactly do you propose to introduce probabilities into the analysis?
Another major difference with Tenembaun is that the IRM considers relations, not interactions (or multiple relations **of the same type** between two actors)
Point 1 is reassuring :)
I think, there an upward trend. Why don't use the ACF plot like [this](http://coolstatsblog.com/2013/08/11/how-to-use-autocorreation-function-acf-to-determine-seasonality/) to check for seasonality?
Thanks, Mr. Hyndman. I was able to find the solution to my problem and posted the solution above.
I think that the word you're looking for is *forecasting*.
I don't see how variance stabilizing can be done first.  You need to see the residuals from the model first to see if the residual variance is changing with time.  Then looking at the residuals might suggest how to change the model or stabilize the variance.
pH, ionic strength, and SAR all sound like they're not categorical at all. And are you trying to predict what kind of soil is observed based on those measures? Given your last sentence it sounds like you haven't even related your response variable.
Skewness is zero for the $t$-distribution, so this particular moment is not informative for any of its parameters.
It's almost certainly caused by high correlation of your two predictors with each other. You might want to google for the term "variance inflation factor" and "multicollinearity". Put simply, if two or more of the predictors are strongly related to each other the model cannot accurately saying which of the variables is important just that at least one is. Posted as comment instead of answer because I don't have time to explain in detail.
Should the first equation have an error term as well: $Y_{ij}=β_{0i}+β_{1i}F_{ij}+e_{ij}$
X would be a number of 60 different currencies, expressed X/SDR. Is that what you mean? The correlation should be pretty much the same, no matter if take absoulut values or relative, right?
This approach is called [centering](http://stats.stackexchange.com/search?q=centering). One of its applications is to turn the regression model's intercept into "predicted y when x is at average," making the intercept a bit more interpretable.
Thanks for your reply! I want to figure out the independent effect of a cognitive ability on a psychological state whilst controlling for certain personality traits. i am using hierarchical multiple regression to figure out how much additional variance of the psychological state is explained by the cognitive ability on top of some traits in personality. I have to meet the assumption that the residuals are normal.
@Brandon: No kidding. This is best exemplified when I try looking for recent events or upcoming events.  More and more frequently, I have to enter the current year or even the current MM-YYYY (or do advanced search) in order to get the relevant pages.  It's a tradeoff between link structure and freshness and Google gets it wrong without more help from me.  In fact, that has led me to Bing a few times, just out of irritation with trying to ignore old pages.  The same is true on SO: first answers often seem to get more upvotes than later answers, which may be more correct.  :)
A search for "Wilcoxon Mann Whitney" on this forum yielded 158 results. I suggest that you need to search for an answer before you can be sure that you have a new question.
Brglm has worked for me to address separation. E.g., pic.twitter.com/vOQgwJDL0T You can get brglm's technical background, Kosmides 2007 Bias Reduction in Exponential Family Nonlinear Models by googling it.
Email the authors for a reprint or buy with USD25 from http://biomet.oxfordjournals.org/content/73/3/543.full.pdf+html
I'd have assumed that a "standard deviate" was someone who browsed not-very-wierd porn sites, but I'll take your word for it.
Applied Survival Analysis by Hosmer and Lemeshow is good; if you use SAS, then Survival Analysis Using SAS by Allison is very good and clear.
Minimizing within-cluster variance automatically maximizes between-cluster variance.  Although this duality does not necessarily hold for other measures of similarity/dissimilarity, it motivates the generalizations.
@Scortchi It seems that you misunderstood me. I want to calculate power of the method which switched to given the sample size calculated from the proposed method.
Yes, just an oversight.
You might start with M. Drton, B. Sturmfels, and S. Sullivant, *[Lectures on Algebraic Statistics](http://math.berkeley.edu/~bernd/owl.pdf)*, Springer, 2009.
I'm going to view your question from a practical stand-point. For decision and comparison between ways to measure errors and residuals, you can refer to [this wikipedia post](http://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics).

In practice, you need to understand your 'penalty' for each deviation from forecast, up or down. Your error sounds very low, so pretty close to forecast. You should also consider using additional variables of underlying factors to improve the forecast. Perhaps the city population is influenced by unemployment rates, number of schools and hospitals, etc.
@Aksakal: the function is certainly not linear (anywhere), but, as I said, "one can possibly use a linear approximation of f without an immediate disaster"
@csgillespie: I was just amazed how quickly people jump in and answer!
All your answers so far (12 today) consist of little more than links to/recommendations of software/publications listed on your website. You might like to read the section in the help pages on [self-promotion](http://stats.stackexchange.com/help/behavior).
I found this article via blog of @Epigrad: http://confounding.net/2012/03/12/thats-not-how-the-law-of-large-numbers-works/.
The simplest way would be to take the average of the two percentages. Or a weighted average if you want to weight one more heavily. Without more information on your problem, it is hard to give a better answer.
*Every* difference is "almost 0"!  If the outcome variable is weight gained by people and it is measured in pounds, then 0.00017 is small indeed, but if it is measured in millions of pounds then 0.00017 is enormous. This question therefore has no meaning until a context--what is being measured in the response--and a unit of measurement are provided.
@AndyW I have already accepted the duplication and edited my question to highlight the relevant parts, since my question is duplicate of a sub-question in the former one.
The ROC curve allows me to see the behavior of my model in specific risk regions, for example in the area with sensitivities < .20. Regarding my question, what I would like to check is if in this interest zone, the behaviour with the test dataset is similar to the train dataset. I don't really care what happens in the region with sensitivities > 0.20.   I would like to know if this comparison will tell me this.