tttjjj/elt_bench huggingface dataset:
Ya, realized what I was doing.  Missed that whole exponential thing.  Thanks.
A simulation in R with varying n1, n2, s1, s2, and population difference would make a nice exercise. Anyone?
@whuber I am trying to find a distribution for a set of sequences (each sequence has a different length but finite). I am using edit distance to calculate the distance between each sequence and put it in a matrix. Now I want to see how can I find a distribution that explains the distance between sequences.
Thanks for the great insight! My only concern is that the paper shows that for zero population correlation, the sample correlation is also zero.
The sample covariance is also biased, at least in the simulations.
If one is using probability sampling, Sampling theory will provide an estimate of the parent populations mean and, more importantly, a variance estimate which one can insert into the weighting. The advantage of applying GLS theory is you have precision estimates. Baynesian regression can incorporate prior beliefs, but again, a variance estimate on your beliefs is required.
Maybe there is some feature that almost entirely guarantees a negative?  Are you talking about the separation on the red line or the other ones?
I am using MATLAB and generated uniform, normal, and exponentially distributed variables using rand, randn, and exprnd, respectively. This approach is the one I used.
some more context might help.
Thanks, that's a technique I could potentially use. It seems like it's been implemented here: http://infohost.nmt.edu/~borchers/ldlt.html
(+1) I was about to answer the same :-) And here is a [case illustration](http://www.stata.com/support/faqs/stat/power.html) with Poisson regression.
@egbutter It is my understanding that ML fits should be used for model selection and a REML fit is appropriate for the estimation of parameters.
For example [this](http://stats.stackexchange.com/a/70910/3277) answer covers it.
Applying terms like "nominal" and "interval" is intended to *rule out* certain forms of analysis. However, which analysis may be appropriate for a dataset is not entirely "deteministic" or obvious, because it depends on the purpose of the analysis, a conceptual understanding of its variables, and even the statistical relationships among those variables. This richer understanding of the typology of variables suggests that a more nuanced answer would be of greater help.
OK, thanks for your comment.
I am not an expert in computer vision, but I would bet there are better and faster algorithms to do that.
Nice! Just one point: consider my example 2, which with your notation is defined as $f(x,y)=x\\oplus y$. We cannot find any function $g$ such that $f(x,y)=g(y)$.
It's questionable whether one should do post-hoc power analysis at all. See e.g [this article](http://effectsizefaq.com/2010/05/31/what%E2%80%99s-wrong-with-post-hoc-power-analyses/). This refers to post-hoc as in after the *analysis* has been done, not just after the sample has been collected.
Any comments anyone? Anyone do lead/lag analysis of related cross sector stock prices?
Ah - that does make sense. They're weakly correlated, but that might be making a difference. I'll try further downsampling to see if I can remove these effects.
a standard statistical approach in this setting is not to look at the distribution of visitors but rather to look at the distribution of _queueing_ visitors given the number of servers. There is a whole field called queueing theory devoted to this. In the past I've used some of the simple results to get ball-park estimates for wait times etc given arrival frequency at busiest period and number of servers.
Thank you so much Rob.
I don't get the last formula.

Why is E[S] = E[E[S|L]]?


And how do you get E[$\\beta$L] = $\\frac{\\beta}{\\gamma}$?
Not so fast with the repeatability argument... First, the experiment that is repeatable is the querying of the machine not the sun going nova The truth of *that* is the fixed but unknown object of inference. The querying experiment can certainly be repeated, and if it were for a few more times the frequentist strategy could easily seem reasonable.
Criteria will be 10% difference in probability of good vs bad outcome. Or lets say since it will be logistic regression, odds ratio = 2. alpha= 0.05, power=80%, I do not yet know what the pooled variance on the continuous variable is but let us assume that the standard deviation is 7mmHg.   Sequential analysis would be good but the final outcome is two  years after the measurement is taken.
"equivalent" means the linear regression results of the two methods (coefficient estimates, etc.) will be more or less the same. In other words, they both arrive at the same relationship between X and Y.
You’re right @whuber and this is not so obvious, just "usual". I has the feeling that the OP was familiar with this, I added a link to a very similar question, I hope this is enough — if you think that it would be better to rewrite it in this special case let me know.
@suncoolsu To nitpick: If you bootstrap, then you *hope that you* create a world similar to asymptotics. ;) Your point is well taken though; the bootstrap works incredibly well and has a pretty solid theoretical foundation at this point. But in a complicated latent variable model like SEM I'd be skeptical of any analysis based on 20 honest to goodness observations, personally.
How accurate do you need the inversion to be?  What programming capabilities do you have?  There are many, many algorithms for this, ranging from continued fraction approximations through numerical integration or even just interpolation in precomputed tables.  You can get some sense of the nature of the techniques from http://stats.stackexchange.com/questions/7200, which discusses computing the Normal CDF; some of the links and references in those answers probably include methods to compute the inverse CDF, too.
I re-read your post and, again, I think I'm misunderstanding you. You're just looking for pairwise correlations, e.g. the correlation between P4 and P5, so forget regression. Could you maybe clarify what you mean by "local" and "global"? What's wrong with just taking the correlation between time series? It doesn't matter that the series themselves are nonlinear, it only matters that their relationship is linear. Or you could use distance correlation to measure statistical dependence.
The limit is taken at the last expression, which is not equal to unity, and it is due to the limit operation that we arrive at unity.
Dan, thanks for you comment.  Would this method allow for the order of events?  For example people who talk to the shop assistant first then touch the product might be more likely to purchase than those who touch first and talk second.
So how did you test whether the estimates improved? Paired t-test? It seems to boil down to the mean change being positive and the median change being negative... Can you go into detail how you set up your analysis? Did you control for the baseline (first round) in addition to looking at the differences?
Cleveland presented a hierarchy of graphical schemata. Although at the top are side-by-side lengths (as metaphors for comparing absolute values), next in the hierarchy are (parallel) lengths that might not share an origin. That is the case with stacked bar charts, so it's not a crippling limitation. Moreover, the SBCs enable ready comparison of *proportions of a whole.* That is not the point of a dot plot, nor do dot plots seem to do that very well. I'm suggesting, therefore, that your (unsolicited) comments about the SBCs may be erroneous and mar an otherwise fine answer to the question.
+1 There's no reason to check general dichotomous distributions because they are all scale and location versions of the Bernoulli: the analysis for the Bernoulli suffices.  My simulations (out to sample sizes of $10^{1000}$) confirm the $\\chi^2_1$ result.
The parameter in the question wasn't eight but ten ... but on the other hand, in my book that's a good thing to change, since it doesn't go as close to just giving the answer.
I actually interpret the response there as saying it's not the case in general, not that it can never be the case. But in any case your argument is looks wrong. You used the same $V$ in $Y_1$ and  $Y_2$. You should have different $V$s.
+1, there are some interesting ideas here. I took the liberty of making a few edits for smoother English & greater readability. I believe I haven't changed anything substantive. If you disagree w/ my edits, please roll them back w/ my apologies.
Age ranges from 18 - 55. So, I just randomly made groups like 18-25, 25-30, 30-35, 35-40 and 40+. Please do suggest an alternative method to study whether height and weight differ across age groups.
You don't prove that $F=T^2$, you only prove that $F$ has *the same law* as $T^2$. By the way you make a mistake: $\\sqrt{F}$ has not a $t$-distribution since it is distributed on positive numbers.
[Van Dam et al. (2003)](http://arxiv.org/abs/quant-ph/0307125) might be of interest.
Yes, unless you can test entire blocks of variables and remove a whole block when P is very large.
"You have just come across expected values"... by what path? Is it by Mathematical Statistics where they are defined as integrals/sums that bring together the probability density/mass function of a random variable and the support of the random variable? It appears not, since you look at the expected value symbol and yet you do not see an integral or a sum. So how did you "come across" them?
There is [Guile-numerics](http://www.nongnu.org/guile-num/) which provides bindings to [GSL](http://www.gnu.org/software/gsl/) which includes [functions](http://www.gnu.org/software/gsl/manual/html_node/The-F_002ddistribution.html) for the $F$ distribution.
Here's a link: http://repository.cmu.edu/cgi/viewcontent.cgi?article=2059&context=compsci (the kernel part starts on the bottom of p.5). By the way, thank you @whuber for being patient with me. Even if we don't figure it out, I really appreciate it.
Note that ``margins`` can be used to calculate the marginal effect at the mean of each x variable: ``margins, dydx(*) atmeans`` . However, this might not be quite meaningful for qualitative variables. More generally, with the ``at()`` option you can fix the x variables at any arbitrary value.
Could you be a bit more specific about the values you have available and the values you want to be able to output.  From the current information it is hard to tell what you need.
It's not particularly geeky - it is a Spanish thriller (and like all thrillers has some implausible plot lines to move the story along and resolve it at the end) with a few logic puzzles using numbers.  If you want to solve the puzzles yourself, a pause button might be useful.
@Placidia I've put my illustration of that last point in an answer below to serve as an adjunct to your (very good) answer.
Hi @stask Good addition to my comment. I have both Agresti and Long, and somehow never thought of either of them as related to particular software. But I think both books are very good.
`SAS` [LOGISTIC procedure](http://support.sas.com/documentation/cdl/en/statug/65328/HTML/default/viewer.htm#statug_logistic_details49.htm) is what you want. You can also refer to [this paper](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3712506/) and the question about [diagnostics for logistic regression](http://stats.stackexchange.com/questions/45050/diagnostics-for-logistic-regression).
When doing visualization, the question is what facts do you want to show with data, not how to just show data (table is way better than any graphic in the latter case).
In case you have not already, be sure to read over the "Large memory and out-of-memory data" subsection of the [high performance computing task view](http://cran.r-project.org/web/views/HighPerformanceComputing.html) on CRAN, which is loosely summarized in [this question](http://stackoverflow.com/q/11055502/583830) on StackOverflow. It might not answer your specific question regarding GBM modelling, but sheds light on R's capabilities of analyzing larger-than-memory data.
Thanks! I'm reading the SLINK documentation right now, and that is what i needed.
@Charlie I mean http://en.wikipedia.org/wiki/Robust_regression. I am new to this, but I understand robust regression changes the estimation technique, therefore the residuals must be different.
You will want to look into "statistics". You will need to fit a model, validate it, and based on that, you can do prediction. Here's a small example of calculating confidence intervals (http://rpubs.com/RomanL/7024), but I'd like to point out the last picture, lower to panels. This is sort of what you're after. I also made a small example of linear regression, see if you can find anything useful (http://rpubs.com/RomanL/7177).
Welcome to the site, @efreet. CV aims to build up a fairly self-contained & permanent body of actionable information about statistics. As such, would you mind providing a brief summary of the info there & why you think it will answer the question so readers can decide whether it's what they want to pursue & in case of linkrot?
Awesome, thanks for the excellent explanation ;)
For sure, there are many issues where control is placebo. But in modern medicinal chemistry to launch a new (original) drug in market the manufacturer should prove that this drug overcomes traditional treatment at least in some situations (certain disease, or its combination, or the disease with associated physiological pecularities). So more frequently the control is a reference drug (in other words, control group is treated with reference drug). Placebo is used when treatment can be advantageous, but is not necessary.
Hi @Florian.  Thanks for that - and the links as well - much appreciated.
A remark: does it probably make sense to z-transform the different item distributions in order to make them compareable?
I think applied scientists and statisticians would agree that allowing negative predicted counts is *unsound* in practice.
@ttnphns, just to point out, this is for agglomerative clustering.
+1 The third point is crucial: any effort to compute CIs or confidence bands assuming these data are independent will produce woefully inadequate (that is, *short*) intervals. It might help the O.P. to emphasize that the method proposed in the question is definitely wrong: it neglects the (strong) correlation among parameter estimates and aggregates the confidence limits incorrectly. The net result actually looks sort of reasonable (purely by accident and luck) but to my eye it still is not sufficiently conservative.
While the terminology may be suboptimal it is common. The oft-cited reference [Dudek (1979)](http://marces.org/EDMS623/Dudek%20FJ%20(1979)%20The%20continuing%20misinterpretation%20of%20the%20standard%20error%20of%20measurement.pdf) talks about "our three formulae
for the three standard errors of measurement".
And for a non-subject-matter-driven decision tree the sample size required to yield reliable structure will be extremely high unless the signal:noise ratio is very high.
NB the first model imposes a constraint that the minimum or maximum of $Y$ plotted against $X_1$ occurs at exactly $X_1=0$. You'll want to be sure that's sensible. It often isn't, & therefore the first model is ruled out a priori.
If anyone is interested, I also talk about the mistaken idea of using standardized betas to infer relative 'importance' here: [multiple-linear-regression-for-hypothesis-testing](http://stats.stackexchange.com/questions/25690/25707#25707)
Ask around among economists. Look at your literature. It wouldn't surprise me if there were many similar studies, but you don't strike me as having much data at all to get good answers. Before and after is surely a crude comparison; the effects of the crisis were not  instantaneous. (I'm not an economist; I just follow the news.)
I would suggest people add some greater context to their answers as well. Examples might be the same word used differently in different fields (fixed effects [Gelman, 2005](http://dx.doi.org/10.1214/009053604000001048)) or words that have different meanings in different contexts (significance vs. statistical significance [Bushway et al., 2006](http://dx.doi.org/10.1007/s11292-005-5129-7)).
The data-sets are going to be used for supervised classification? are they labeled? If so, you could initilly just separate them in an 80% training set / 20% test set. Train your model on the training sets and see how it predicts on the test sets. See which one yields more precise results. Or do some form of cross validation for the same purpose.
Welcome to CV, and thanks for your answer!
If this is done purely visually, perhaps the nicest name for it would be "data fudging." If you are applying a definite algorithm, and are willing to disclose it in your post, then perhaps more could be said.
@aginensky yes it is not simply random {-1,0,1}, they do have relationship with y, but I am trying to make of a "non-sensible", reproducible example
I got the answer research.cs.tamu.edu/prism/lectures/pr/pr_l7.pdf‎. This will clarify. It was stupid question :(
@EdwardArmstrong: I am not familiar with the literature on NAO, so have no idea how it is usually defined. However, I can imagine that there is some normalization involved, because otherwise the values of the principal component are going to depend on the number of locations you have the pressure measurements from. E.g. if you had twice as many locations, the PC1 you get would be scaled by $\\sqrt{2}$. One way to normalize the index would be to divide it by its standard deviation (assuming that means are subtracted already at each location). Then you would expect NAO index  be between $\\pm 3$.
To fill out user603's answer, if you imagine the eigenvectors as new axes, then the difference in sign is only about whether you read them from left to right or from right to left - they're still running along the same line.
+1: You may want to explicitly say that the likelihood is necessarily Dirichlet, which is why the posterior distribution is easy to compute.
+1 for this answer. You should be able use a contingency table to answer your question.
I didn't actually know any specific ones when I typed my comment (in spite of understanding it wasn't generally true) but it didn't take long to figure one out. I wanted to try to think of a continuous example for which all the calculations were simple. Didn't quite do that (since the median of a $\\text{Gamma}(2)$ random variable involves the incomplete gamma function), but if I think of a simpler continuous one I will probably come back and include it.
Thank you so much Ben, that worked out. I decided to use the time variable as a continuous covariate and fitted the (Day|ID) term, I think that is the best choice,
The [same question](http://math.stackexchange.com/q/295363/15941) has been asked by the OP, and an answer already accepted, over on math.SE
Indeed, it could not have been standardization, thank you! Actually, I just realized it's a coding problem.. which is an entirely different problem.
I was thinking about doing it this way: I can calculate SMA and write down the calculated average means, then I can compare the differences between the means, and if there is no difference, I can be sure that my data is stationary. For example: I have numbers = 1,2,3,4,5,6,7,8,9 and a window = 3, then average means are: 2,3,4,5,6,7,8,9 so the differences between them is always 1 (constant). (When I do avg[i+1]-agv[i] I always get 1 for my calculated means). So should I consider my dataset stationary?
I can't answer that question, you might want to open a new thread for specific details.
after applying weighted penalty for each class it works faster. Going through all the mathematical intricacies of SVM now. Thanks for help
Missed your answer somehow (+1) deleted my duplicate.
As an addition: Kohavi studies the bias-variance-tradeoff in validation in chapter 3 of his [Phd thesis](http://robotics.stanford.edu/~ronnyk/teza.pdf). I highly recommend it.
What about the kalman filter and/or smoother? Where does that fit in?
As long as you don't define what you mean by "want to just 'control' variables" you won't get an answer to your question. From the figure caption in the (unsprecified) publication you are referring to it should be possible to deduce how they "controlled" the covariates.
@bit-question, is this exact citation? Could you please give a reference?
Could you clarify your description of model and outcome? You state in the question that you have a logit or probit model, which models binary outcomes. Yet in your comment, you describe your outcome as "cumulative purchases over time" which to me sounds like count data.
@brandon Now, you can (15 rep needed).
See this [answer](http://stats.stackexchange.com/a/13700/2116) of mine.
Actually, the question exhibits a thoughtful understanding of the problem and does not appear to be looking for an answer to a routine homework question.  Although it's tagged [tag:Homework], consider making an exception here.  In particular, a good discussion of using the Normal distribution to approximate discrete distributions (such as Binomials and Poissons with large N's) would be appropriate and most welcome here.
I think *not defining them as factors* would be far  more dangerous Do you expect the effect comparing no education to primary to be the same as comparing middle to high school and exactly half  the effect of comparing primary and high school.  They are categorical data, not continuous, so you need to code them as such and model appropriately. You could go down the path of ordered factors if you wished, but I think that would add more assumptions to your model
Thanks. I cannot up vote the answer. Looks like I am not allowed to do so.
thnx for you reply, but i think i have used the same formulas, i didn't get what you wanted to add
I've never used Weka, and I wouldn't ask my worst enemy to do regression using Excel, but it appears that you have reverse coded some of your dummy variables. You've put Woche4=0 in Weka, but (assuming it's coded 0, 1) the variable that's used in Excel will be Woche=1.
lejohn has a very good answer, you should vote it up (and potentially accept it if you don't get one you like better)
(+1). NA more generally means that the coefficient is not estimable. This can happen due to exact collinearity, as you've mentioned. But, it can also happen due to not having enough observations to estimate the relevant parameters (e.g. if $p > n$). If you predictors are categorical and you're adding interaction terms, an NA can also mean that there are no observations with that combination of levels of the factors.
Thanks alot! Sounds straightforward. I'll have a go at writing the equations on monday.
As I understand it is possible to reconstruct the structure but it is impossible to assign class specific lineage of nodes. Here is my problem. I have a binary problem (presence/absence) of variable that can be predictive of one of 4 classes (as single and as joint for example combination of 5 variables). I am getting high prediction relying on top 20 by VIMP out of 1500 variables). I have to extract the specific combinations at least some of them
Building on @whuber, and just to check on what you mean by "segregation" - do you actually mean just varying proportions?  For example, there is a higher proportion of Spanish speakers in Los Angeles than there is Vancouver, but I don't know that this would mean "segregation" in the common English usage.  But this seems to be what you are looking for.  Perhaps a better expression would be "varying proportions" of the language groups.
Background: This is for this project: https://github.com/tomrittervg/crypto-usability-study where we aim to emulate an attacker who can expend 2^80 random bit generations, and is trying to match a given bit sequence. We are assuming they will either choose to match the maximum number of bits they possibly can OR that they will prioritize the bits in the beginning and ending of the sequence.  If anyone would like to give involved and help us with the math, send me an email, we'd greatly appreciate it. https://ritter.vg/contact.html
The link no longer works