tttjjj/elt_bench huggingface dataset:
Google has the best search engine in the world.  I bet they came up with good solutions to your problem.  But some of the best tricks may be proprietary.  Now they don't have the exact same problem that you have because they can make long lists.  But they do order them based on algorithms like the what you are looking for.  With Google, Yahoo and so many other search engines out there I am sure there must be a vast literature in open journals that you could explore.
I don't follow your overview. Is it that 2 people mentioned customer service among their reasons & only 1 person mentioned speed, or something else?
It's true that #1 and #2 are mainly solved by some manual mappings of ballots to ballots and of old parties to new ones - I don't see a good way around this (but if anyone has a good idea, I'd be happy to hear). #3 though is independent of the previous two questions and has to do with statistical theory, method and practice. This is where I'm looking for help.
@juampa, data seems alright. funny thing is I added some more data of class 4 in training, and the no. of samples in class 4 is now 300. With this, I get good results in training. Test result is not so good though. But why so?  AccTrain: 0.814126
AccTest: 0.566581

Mean Diag train: 0.894766
Mean Diag test: 0.416521
Confusion matrix training:
  [ 148     0     0    55;
     0    19     0     0;
     0     0    16     0;
    44     0     1   255]
Learn to use google. typing `transfer entropy + R` leads to the following link on top `http://users.utu.fi/attenka/TEpresentation081128.pdf` which contains the code that you are seeking!
I had mistakenly flagged this question as too broad when it should have been off topic. Given that the question is only about code and not conceptually about DiD it would be more suitable for SO
So you need to explain this in your question. What exactly do you measure? Have you captured the number of episodes and the duration of episodes for each individual and PD is the length from the start of the first to the end of the last?
cyborglizard - several people have asked you to clarify this point, and as far as I can see, you still haven't done so in a way any of us can clearly understand (I took a guess for my answer). When you say "*continuous nonlinear independent variables*" what is nonlinear with respect to what *else*? A thing isn't nonlinear by itself. Are you referring to the index order (the order the observations are in) as your 'x'? Why would that be relevant? What variable is represented by that order? Are these observations over time or along a spatial dimension or something? What *is* b5, exactly?
I don't know why, but it did not return the last created object?
The non-binary data is discrete. It is being converted to binary classes using dummy coding.
@Aniko I don't think the OP said that they were handed out in batches.  The OP is assuming one at a time.
A drug may be given because someone has a disease.  You can't claim then that the drug caused the disease; it's the other way 'round.  Also, some diseases may predispose patients to gain weight regardless of their meds.  There may be a way around this but you need to very explicitly lay out all your assumptions, then critically assess them.
*"For a non-normal GLM, the distribution of residuals do not matter"* - this statement is false, the distribution of residuals and thus the error structure is strictly defined by the GLM family: poisson, binomial, ...
Hm, this appears a bit circular: A correlation is calculated over all values, so, each time you re-calculate it on a subsample of the original data, you alter it. This way you might continue removing 'noise' until you end up with two values per subsample.
Consider this posting as a stand-alone question. I would be very interested what gurus have to say about this.
Thank you onestop. We discarded RCB design because the blocking factors we evaluated in simulation studies proved not to extract enough MS for the degrees of freedom we lost by blocking...
with (1) I have no idea what you're even asking - e.g. $n$ isn't multiplied by $\\alpha$; with (2), what's after the minus sign is an $\\alpha$. Are you just asking where the $\\log x_i$ comes from? The central problem simply sounds like you're unfamiliar with the mechanics of taking derivatives --- e.g. can you say what $\\frac{d}{dx} a^x$ is? If that's the case, it would be a problem better dealt with on math.stackexchange.com; failing that, at least proofread and edit your question carefully to be clear about what you're saying. Can you write $L$ and hence $\\log L$ down before simplifying it?
@Alexis The answer is intended to show instantaneous rate of change, since the OP is interested in finding fluctuations in a variable over time. Added some clarification in the answer.
can you explain the coefficient thing a little more elaborate for beginners? I feel your approach is worth a try because if I difference logs in my case (growth rates) the trend flattens but seasonality becomes strong. Thus the simulateneous approach seems worth trying. But what do I do with the two coefficients? in particular i mean the dummies...
Unless $a$ or $b$ are small and unequal, the Normal approximation will be accurate to better than $\\pm 5$%. Is that fast enough?
I haven't time right now to do so in detail but look at `glm` and search on logistic regression.
Remember the Jacobian when you change variables.
(+1) thanks for the great answer
@PeterFlom The KW test assumes that sampled populations have identical shape and dispersion, although in most case little departure from those assumptions will not affect the results. When parametric assumptions are met, the test is $3/\\pi$ as powerful as one-way ANOVA. Regarding rank-based test statistics, some studies suggest, however, that varying degrees of skewness may inflate nominal type I error rate, see, e.g., [Fagerland and Sandvik (2009)](http://www.ncbi.nlm.nih.gov/pubmed/19247980), or some [other](http://goo.gl/umM0Nn) [references](http://goo.gl/8BGbjH).
I was asking specifically about the item ranks, but if there's a good way to show inter-rater agreement other than something than Kendall's W, all the better too.
@darkcminor: You might previously have calculated $Pr(A| \\lnot B) = \\frac{Pr(\\lnot B|A)Pr(A)}{Pr(\\lnot B)} $ in much the same way.
Anybody "running experiments" should take the time to study statistics!
Hi Dave, thanks for your reply. Though I do not have access to the book, I visited NLTK website. "Classifier" or "Categorize", etc, keywords are searched, but no relevant pages show up. Could you give more specific hints on using text classification using NLTK?
**Intuition:** Gamma$(n)$ distributions arise as the sums of $n$ independent Exponential distributions, whence it is immediate in this context that $X+Y$ will have a Gamma$(a+b,\\theta)$ distribution provided $a$ and $b$ are both positive integers.
Having no model selection in introductory courses isn't a necessity.  If you're considering changing a course consider that as a good place to start.
Hi @mafbarth! If you use the standard error for generating the random values, you are generating new **mean** values and not original measurements. Without knowing the sample size, it is impossible to derive the standard deviation of the measurements. So: wether it is appropriate to use the SE instead of the SD is a matter of your question.
$B^{12}e^{t} = e_{t-12}$
Another good read is Griffiths and Steyvers "Finding Scientific Topics" (http://psiexp.ss.uci.edu/research/papers/sciencetopics.pdf). This paper introduced the Gibbs sampler for the LDA model.
Just a clarification: if $X_1,\\ldots,X_p$ are the original random variables, then $\\sum_{i=1}^p\\lambda_i=\\sum_{i=1}^p{\\rm Var}(X_i)=tr(\\Sigma)$ where $\\Sigma$ is the covariance matrix.
super useful!! thank you so much for posting.
It seems that a disadvantage of this implementation is, that one cannot define a bn-architecture on his own.
Caveat: the Nurses Studies were observational designs also. While there *are* strategies to strengthen causal inference, the data from these studies are also based on correlation.
Turns out our data is non-convergent (discovered after eventually finding a copy of SPSS to run it through), so R spins out of control unless `clogit` is supplied with the argument `method=c("approximate")` I also found this video helpful: http://www.youtube.com/watch?v=be-M0tMMKyU
**Hint**: I think you are confusing taking the expectation of a certain quantity with finding the probability over a particular region of interest bounded by a function.
Thanks @Charles that makes sense. What does it mean then when people talk about "non-parametric MLE". That phrase doesn't makes sense at first glance. MLE always estimate a _parameter_ of the distribution, right?
For your multiple regression, yes, but with all four independent variables. If any of those do not improve the model of `salary` significantly, you may wish to draw the insignificant paths as dotted lines. I recommend **not** using such a figure for bivariate correlations like Pearson's $r$.
I don't quite follow the impetus behind this. If you have a method to extract & classify "spider" & "cat", & these are perfectly nested w/i higher level categories, why not use that method? Then you could feed the output (s & c) into a simple rule-based algorithm which will add the overarching categories. This seems fairly straightforward to me, although I'm not an AI person.
I suspect that perhaps what you are calling probabilities are not actually probabilities, or at least they might not be *conditional* probabilities conditioned on the known information.  For instance, I'm pretty sure the probability in English of "the" being the third word, given that "one of t" has appeared, must be way higher than any probability in your supposed ranges.
I don't know if it's possible to give some general advice on this but in any case it's known as the “class imbalance” problem. Using those keywords should allow you to find a lot of relevant material on this site and elsewhere.
Ecah plot was split into two (creating two subplots per larger plot) and Treatment Two randomly assigned, such that one subplot per plot received level "A" of Treatment 2 and one subplot received level "B".
I copied mine from the original publication.
When I last looked at detailed traffic data (I recall it was annual summaries by road segment throughout the state of Washington in the 1990's) I performed extensive testing for evidence of deviations from Poisson distributions, but failed to find anything significant. Strong influences of covariates ought to create non-Poisson, overdispersed data. The absence of such behavior suggests it may be rewarding to start any analysis with the data that are at hand, before going off and hunting down speculative covariates.
This looks like a sensible approach -- deal with point 2 separately.  What kind of test can I use?
thanks @Gavin. I should have searched better.
+1 for a constructive answer. Automatically generated and attractive design are _not_ contradictory; it is just much harder.
Can you please expand on what you mean by 'per subject level'. Are you interested in individual differences?
thanks, the papers I had added to my confusion by not being explicit between Sample and population estimates. However realising the cancellation of n or n-1 respectively makes everything clear.
Logic stays the same. If you observe that two monsters $n$ and $n'$ do not drop the $m^\\text{th}$ object then the probability of that happening is: $(1-p_{nm}) (1-p_{n'm})$ Include this probability as well in the likelihood function and maximize the likelihood function to estimate $p_{nm}, p_{n'm}$. You can also get confidence intervals for your estimates via maximum likelihood theory.
What do you mean by error in lag.  If you do crosscorrelation the highest lagged correlation might be taken as a measure of how one time series lags behind the other.  But what are you looking for in this?  Is it that you want to address the situation when two or more adjacent lags have high crosscorrelation values but the highest is only highest by chance?
I'm actually looking to assign a score to each classification, so that I can use the score to order the documents. For example, it would be useful in a recommendations setting where I want to classify documents into those that the user will like and those that the user will dislike, then use the classification score to order the recommendations.
Running an initial look at the relationship outside of the formal analysis will get the d.f. wrong, resulting in inaccurate confidence interval coverage.  A good regression text will describe how dummy variables are constructed.  Good software packages do this automatically if you make them sense that treatment is a categorical variable.
A Chow test could help you with (1). It tests whether the coefficients are equal in a model estimated on two different sets of data. It requires you to have specific subsets in mind, but it looks like you've identified the most likely break points. You could also compare the AICs of the two models if you know their likelihoods.
Oh, I am really sorry! I completely forgot to say that I did that. I am using the Tobaco and Alcohol data set for UK, so my whole MSE expression included: `X = df[["Tobacco"]].values
X = sm.add_constant(X)  # add the intercept term
reg = LinearRegression()`
I have a time series for each customer. I would like to find changes in the habits of individual customers.
do you have time to event in hh:mm or hours?
@ankc: Reducing the sample size doesn't fix any deficiencies in your model, but only hides them. Why would you want to do that?
@shn it is a common mistake to think that you are not learning and that there are no parameters in such a method. As long as you use the training data, it is learning. What you are actually doing is using the whole training set as your "learned parameters", so when you are saving it for later use you are actually "training" (this is why NN is often more prone to overfitting - it actually has a lot of "parameters"). If the predictions you make are dependent on the training set, it is learning. A case with no training would be if you would make predictions WITHOUT using the training set at all.
I don't disagree with your facts, but your English summaries do not seem to be easy to read for a non-statistician: 1) "the ratio of the hazards of buying a couch at any time for an individual who has a cat relative to an individual who has not a cat" ??; 2) "the hazard of buying a couch at any time t for a person who has a cat is superior than that of a person who does not have a cat" ??  Remember, this question is about how to phrase in English for a non-technical audience.
Thats great, thanks. I'm actually looking at sentiment analysis of text documents - classifying them into "positive" or "negative" - there is a lot of work published where people alter the feature selection in some subtle way, with say 1% accuracy improvement - and write a paper about this. I'm wondering whether in many of these cases the authors have failed to give evidence for a statistically significant improvement in classification accuracy.
this [question](http://stats.stackexchange.com/questions/5782/variance-of-a-function-of-one-random-variable) and its answers can be of interest.
I think I understand. I believe `outcome{1,2,3}` are actually supposed to be indicators of the outcome represented by `pi`, not the outcome measurement itself--`pi` captures that. Then the parameter estimates for them are the outcome specific intercepts (thus, exclude the standard intercept with `+0`). Then to model the correlation between the outcomes, you need allow the intercepts to vary randomly and correlate with a term like `(outcome1+outcome2+outcome3+0|individual)`. This is a comment because I am not comfortable enough with the procedure to make it an "answer".
I'm sorry but I think I did not really understand what you meant. You mean to use the residuals as the target variable or as a new predictor replacing this annoying parameter?
@Andreea, I have added an answer to address your comment.
One of those links you reference shows how to deal with the fourth root.  What part of it are you having trouble adapting to the reciprocal square root?
+1 However it should be noted that the Mann-Whitney will give a CI for the population Hodges-Lehmann location-difference (median of the distribution of cross-group pairwise differences, a.k.a. the [*pseudomedian*](http://en.wikipedia.org/wiki/Pseudomedian) difference). It would be a proper CI for the difference in population *medians* if the two population distributions have the same shape and spread -- but then it's equally a CI for the difference in means, modes, tenth percentiles, midmeans or whatever you like (as long as the population quantity exists and is finite).
A "scale" which is multi-factor in its structure should rather be called a "battery". Anyway, it may not be expected to be homogeneous, i.e. high alpha.
you could try INLA or the spaMM package for R.
Welcome to the site, @Cherese. At present, this is more of a comment than an answer. Would you mind elaborating on this a little?
This is not true when the p_i approach zero as i increases.  Otherwise, you have just proven that the Poisson distribution is Normal!
"Hardly publication quality"???? I realise that it isn't perfect - the phrase "...should you get you started.." covers that bit. But with a little additional work, i.e. axis labels, I would say it's fine. BTW, what journals do you publish in?
Ah yes, making a factor of 700,000 unique values is a recipe for disaster. It is safe to ignore the warning, but I will adapt the code in the next release to prevent it. Current behavior is that character variables are simply copied, and not imputed. This seems appropriate in your case.
Hello, thanks so much for this answer. Then do you know if stata or spss (or even matlab or python) had an appropriate package
I would think that a random forest would handle this well.  Are you sure that your RF settings are appropriately set?
A couple other notes: in practice I usually confirm convergence by comparing predictions from one half of the trees to the other.  As far as overfitting, it's more of a function of what you're trying to generalize to.  They aren't going to overfit if you are training on a representative sample, but that's seldom how it really works.
This doesn't read like a combine-the-distributions question: the formula for `icorr` looks like it was fit to some data. Are you perhaps trying to compute prediction limits?
+1 This actually answers another question that I had. But is this test set up to reject whether $\\mathcal{L}_g>\\mathcal{L}_f$? Or can the test work in either direction?
Don't let the wording trick you. You matched individuals by some _independent variables_ (such as age, gender and whatnot), but your outcomes are likely to be dependent within matched pairs of individuals. Conditional (fixed-effects) logistic regression is a possible simple way to go here.
Much of this reply assumes that $10^{-4}$ is "small," but @John rightly points out this is a matter of units of measurement (and therefore is meaningless).  The large SD (compared to the mean) shows that there is *considerable variation* of the individual slopes compared to their average.  This might or might not be consistent with the true mean being zero.  Finally, we cannot assume these slopes have a common sampling distribution; in fact, that's unlikely.  Thus the SD actually tells us little about what's really going on.  We need more information to say *anything* here.
Could you elaborate on this a little bit more? Thank you.
Why aren't you using a two-sample t-test on the individual observations? That seems to be what's recommended by the author of the web-page you cite.
How's that for a more lucid explanation?
(1) What is the intended relationship among $a$, $v$, and $\\Sigma_{A,B,C}$? (2) What do $A$, $B$, and $C$ mean?
http://www.quantitativeskills.com/sisa/statistics/corrhlp.htm has a power analysis.  If the true population correlation was -.264, and you had a sample of 160, then you would have a power of 92% (i.e. you would reject null 92% of the time).  Play around with this and see what a reasonable sample is given the size effect you expect.
PCA can definitely be useful for graphical examination of your data set. By drawing scatter plots of the first few principal components, you may be able to see if samples from the same group cluster together. Regarding finding a good separatation of the clusters, this is the problem that _discriminant analysis_ seeks to solve (see for instance [this Wikipedia page](http://en.wikipedia.org/wiki/Linear_discriminant_analysis))
@Roman Luštrik, I am not sure what graph you need. How should I build it? The only graph that I have is `y` as function of `x` (it ignores values of other variables). It looks like `y` linearly depends on `x` with a lot of noise. The second problem, I think I am not allowed to post the data on line, since it is property of the company that I am working on.
From an analytical perspective HB analysis = multi-level models. However, the term multi-level models is used when you have different levels that occur naturally (See the example of @ars). The term HB models is used when you do not necessarily have different levels in the situation. For example, if you are modeling a consumer's response to various marketing variables (e.g., price, adv spend etc) then you may have the following structure at the consumer level: $β_i \\sim N(\\bar{\\beta},\\Sigma)$ and $\\bar{\\beta} \\sim N(.,.)$ at the population level. For references: See the other answers.
I nice online guide that mentions overdifferencing is: http://people.duke.edu/~rnau/411arim2.htm
I need to show that the sample mean $\\bar{x}$ is an unbiased estimator of $\\bar{X}$.

I have $E[\\bar{x}] = \\frac{1}{n} \\sum E[x_i]$

Now, I need to evauluate $E[x_i]$
So I think the overfitting problem is the classical problem of using too many features with too little data.  The only cure for overfitting is to reduce the number of features or increase the sample size.  Finding another methodology will not help.
Substitute for $\\varepsilon$ in $E(\\varepsilon E(Y|X))$, and use the law of total expectation by conditioning on $X$
"The median of the posterior distribution"; distributions have medians.
Try a model with multiple seasonalities.  For example, tbats() in the forecast R package.
I see know reason why values in the test would have to be in the training set.  The idea of classification is to use the training data to get an idea of what the class-conditional densities look like.  You don't get to see every possible value from the density.  I a variable is used in a split ona tree then the split determines which branch to follow for any unseen values as well as those that have been seen.
@rpierce I have added the code sample as requested. I originally wrote it in LaTeX/Sweave, so the lines of code were interwoven with my comments to myself. I used version 1.1-6 of lme4, which is the current version in June 2014.
Your question has an answer, so you don't really 'own' it any more - it contains work by other people (an answer) and you don't get to delete their effort without a very good reason. If there *is* an especially good reason, you should flag it instead and explain it to the moderators.
Nope, your answer was great! I just usually try to avoid accepting an answer too quickly, and haven't visited the site in a while to come back and accept =).
The problem is fomented by some professors. My PhD is in psychometrics, which is in the psychology department. I heard professors from other parts of the department say things like "just report the p-value, that's what matters". My work is consulting, mostly with graduate students and researchers in social, behavioral, educational and medical fields. The amount of misinformation that is given by doctoral committees is astonishing.
@raegtin - I think so.  The GLM weights, $w_{i}^{2}=\\frac{1}{\\phi_i V(\\mu_i)[g'(\\mu_i)]^{2}}$, are equal in both cases, and the link function produces the same logit value.  So as long as the X variables are also the same, then it should give the same results.