tttjjj/elt_bench huggingface dataset:
The mean structure does not suggest either a cubic or quadratic term: they could not accurately reproduce what is shown.  There are several ways it could be handled: allow the slope to change discontinuously at an unknown point (that adds two parameters) or add a (decaying) exponential term, which also needs two parameters. If you want to anticipate the outcome, eyeball the break (it's around $3/4$) and introduce a variable equal to time before the break and zero after the break. Although this doesn't account for uncertainty in estimating the break, it will show you a better fit.
In R: pnorm(26, 15.9,3.6)-pnorm(10, 15.9,3.6)
For those (like me) who might wonder what a "Laplace coin" might be -- it looks like it's another term for "fair coin".
I said at the beginning of my question that I was just looking for some advice on where to start. I'm not looking for a magic box. There is no such thing. I would just appreciate some advice from someone who has more experience with this kind of thing than I do, so I don't spend too much time going down a dead end.
Since this seems to be routine bookwork please add the self-study tag (remove one of your 5) and read its tag wiki info ... http://stats.stackexchange.com/tags/self-study/info ... modifying your question as needed.
I believe it's not possible to get p-values that low on the Wilcoxon test unless the sample sizes are pretty large (or at least of moderate size--$10$ or greater--and the differences are as extreme as possible). But given that the differences are stated in terms of p-values and not effect sizes, why and how would any such conclusion depend on sample size?
There is no problem in using either of the two methods. Because you have scaled data, it will be advisable to use non-linear regression.
If we're *really* interested in (unnormalized) *sums* of independent random variables (on the same probability space), we can say something quite a bit stronger. It is a theorem that $\\sum_{i=1}^\\infty X_i$ converges in distribution *if and only if* it converges almost surely. Now, if they are iid, then the Kolmogorov three-series theorem implies that $X_i = 0$ almost surely and so the limit distribution is completely characterized.
You sampled without replacement (otherwise there'd be a lot more than 15 combinations of sample sizes of 4), and, as a consequence, the individual draws are not independent.  That messes things up, as one might suspect.  To see this, consider drawing 6 times from (1,2,3,4,5,60) without replacement; all your samples are the same, and the sample variance using $n$ in the denominator equals the population variance every time.
+1 for the thoughtful considerations.
You have very many, & very basic questions here. What you need is to take (possibly several) statistics courses, or, if you are not a student, get several statistics books & work through them. Even if all of these questions were answered here, & the ideas behind these issues were explained, I suspect there would be other topics you don't know or misunderstand that you didn't think to ask about. Providing all of the relevant information here is, in effect, a statistics course / textbook. You should work w/ a statistical consultant.
@NickStauner if you consider the nodes to be columns or rows in this example this may be considered an adjacency matrix for a bipartite graph.
If the algorithm works properly (which I expect from a Matlab Toolbox), you can "strip" the imaginary components and leave only the real values. However, I strongly suggest you to learn what PCA actually does before claiming that your results are ok! There are plenty of ways to check that your results are coherent with what you wanted to do if you understand what PCA does.
(+1) This would also suggest a good graphical comparison would be a QQ plot of before vs after (or potentially both super-imposed on a single graph vs a theoretical distribution if a comparison to a theoretical distribution is applicable).
Nice to see you back...
Finally, sorry if I sounded grumpy on [so]. It wasn't intentional. I just forgot that you can't vote to close as OT & migrate to [stats.se]. You have to leave a comment as to why but I'd already left the first comment. Don't choose a SE site for your question based on the number of eyes that will see it. Choose the most appropriate venue. There is nothing wrong with promoting your question on [stats.se] to get more eyes, you could even post the link to it in the R public Chat Room on [so]. Crossposting or posting OT questions dilutes the resource if we have too many, hence close votes etc.
I have checked, Andy: convolving the discrete solution with a normal distribution does not change the skewness.  When you give that normal distribution a standard deviation around 0.57 or greater, the result is unimodal.  Like the underlying discrete distribution, it continues to have zero mean, zero skewness, and to be asymmetric.  Mixing this in with a standard normal distribution amounts to a controlled movement of mass between the standard normal and the discrete distribution: that might fulfill your request for a "stereotyped" method.
Just because your data are spatial it doesn't necessarily follow that they are spatially dependent. For an explanation of spatial dependence see http://stats.stackexchange.com/questions/18406/what-is-the-difference-between-spatial-dependence-and-spatial-heterogeneity.
Fair enough, we could debate about whether or not the algorithm is statistics-related for a long time. The reason I mentioned that is that I know some researchers that work on this algorithm, and they never worry about its statistical properties (consistency, recovery, ...). The community revolving around it seems to be more of an optimization community than a statistical inference one.
It looks like there are two versions of the QR factorization. I was taught The "skinny" version where A is mxn,Q is mxn and R is nxn. See http://www4.ncsu.edu/eos/users/w/white/www/white/ma580/chap3.3.PDF
Are you wondering about any of the statistical issues associated with the Weibull distribution, or fitting a distribution via maximum likelihood, or are you just wondering how to get this done in R? If the latter, this Q would be off-topic for CV (see our [FAQ](http://stats.stackexchange.com/faq)), but on-topic at [Stack Overflow](http://stackoverflow.com/). If you're interested in statistical issues, please edit to clarify, if it's just programming, flag your Q & we'll migrate it for you (*please don't cross-post, though*).
@Manoel: Outcome 1 has 209 data points; outcome 2 has 72 data points. Yes, the samples are independent as they are either predictors for outcome 1 or outcome 2 (mutually exclusive).
Each person are measured once; Randomly assigned to Drug. Data on drug includes person that has consumed it and not consumed it.
Rotation is being done for the purpose of interpretation of PCs. If you have such a purpose that means that you treat PCA as factor analysis, a latent-variable analysis, and not just as data compression technique. You should rotate every solution from 7 thru 13 components and interpet; leaving in the end the most interpretable one.
Karl already summarized everything perfectly, I just want to add that $EF(x)$ is understood as a expectation of random variable $y=F(x)$, where $x$ is the random variable. If $x\\sim F$, then $F(x)$ is uniformly distributed in interval $[0,1]$, so $EF(x)=1/2$, for any $x\\sim F$. Definitely not the kind of definition you would want to use :)
The report about your analyses was needless, really. The title and "Question" would suffice. `Limitation`? No limitation: you may do whatever you want. `Consequences/differences`? There will be, some. What kind of? Please try it, to see! If you come across differences which you cannot explain yourself - then ask about these here.
How do we know the functional form for the edge length distribution? do you have a derivation/reference?
@Flounderer For positive integers $n$, $\\binom{-n}{k}$ actually does have a combinatorial interpretation and it *should* be nonzero when $k\\ge 0$. Even when $n$ is nonintegral--even when it is a complex number!--this has a clear definition and makes sense: it is the coefficient of $x^k$ in the Binomial expansion (Taylor series) of $(1+x)^n$.
@chl: No. Frequentist only.
Thanks a lot for quick reply. So if I have a variable having entries "good","better","best","worst",then I would create 4 dummy variables or  3 dummy variables. Because in case of linear, it will be three.
Is the dependent variable also on a 7-item Likert scale?
Actually, it's not homework.  It comes from a probabilistic approach to cache eviction in an "LRU" cache.  Pick 5 items at random and evict the least recently used.  I'm just trying to get a sense for how much worse that is than choosing the least recently used item exactly.
"*its for my own studying*" -- which thereby still falls under the [self-study tag](http://stats.stackexchange.com/tags/self-study/info). Would you please add that tag?  (NB It doesn't actually use Bayes' rule)
@whuber, I believe this *is* the OP.
Here is another proof: Most Bernoulli random variables have median $0$ or $1$. The estimate from $n$ trials depends only on the average values of the estimator on the vertices of $[0,1]^n$ with $k$, and the weights of these average values is a polynomial in $p$ of degree $n$. If this is an unbiased estimator, it must have average value $1$ for any $p \\gt 1/2$, and there are more than $n+1$ such values of $p$, so this polynomial must be constant... but it must be $0$ on lower values of $p$, so it can't be unbiased there, too.
It is not exempt. What I was trying to suggest that one would withhold data ( as you suggest ) aand then make forecasts from a number of origins. The withheld data would never be used to alter the model or the paramers as that would be "cheating" .
Another problem with the passage you quote is that there is no clear definitions for the terms "random" and "haphazard".
How would I get the standard deviation for these functions? How would I use the dispersion?
(+1) Very nice answer.
The following CV questions also discuss the relationship between GEE & GLiMMs: [What is the difference between generalized estimating equations and GLMM](http://stats.stackexchange.com/questions/17331/); [When to use generalized estimating equations vs. mixed effects models?](http://stats.stackexchange.com/questions/16390/)
I don't see any particular problems it would raise. But there are the *usual* problems. E.g. you'd have to check collinearity.
I don't understand what you mean by "punctuation", in English punctuation refers to things like periods (.), commas (,) etc.
It's not going to lead you instantly to the answer, but did you notice that $X = UV$?
My understanding is that you need a finite mean to for a population L-*moment* to be defined, but not for the estimands corresponding to all other L-estimators. For example, the sample median is an L-estimator, though it isn't an L-moment.
I now see your point, the first part of Jon's suggestion is correct, and is equivalent to Laplace smoothing; but the suggested implementation (while it would probably work) is merely an "heuristic", rather than a proper solution, and isn't equivalent to "start[ing] all counts with 1"
It looks good. i would have loved it if it would accept non numericals. It seems one has to transform their data to a numerical list first.
Are you aware of [Fieller's theorem](http://en.wikipedia.org/wiki/Fieller's_theorem)? The case of $log_{10}(r)$ is also discussed on the Wikipedia page.
With loess, you can adjust the amount of smoothing, did you try different amounts?
Then why not say so up front so as to give a completely correct answer from the get go, or edit your answer now that the omission has been pointed out?
Well, the acceptance region and the rejection region should form a partition of the range of $X$, don't they? If you ignore $\\{x: \\varphi_k(x)=k\\}$, then the union of the acceptance region and the rejection region  will be a proper subset of the range of $X$, not forming a partition.
The fact that you only have dummy variables for predictors does not make any difference.  So this question reduces to references on weighted regression which would turn this into "big list" question - there are so many, where to start?  And as @Adamo asks, we need to know if the weighting is because you are weighting to population, frequency weighting because you have multiple observations of some points, or weighting for some other reason eg to account for variable variance.
See this http://stats.stackexchange.com/questions/52631/conditional-expectation-has-the-minimum-mean-prediction-error/52663#52663
@user1885116 bear in mind that `ns` produces a different basis to the one shown here as it's a natural B-spline rather than the truncated power basis. As for the dimensions of the output - what is your data, how many knots are you using and where are they placed?
Converted to community wiki.
Ok it wasn't that complicated finally. I just modified the 'size' arguments (adding hidden layers) and the response totally makes sense. So I guess I did right :)
Thanks!
Thank you for sharing your data! It was very interesting to look at. I don't have a good answer to your questions, but I do want to suggest some things. For Question 1, you can try adjusting the control parameters in the `LDA` function in the `topicmodels` package. In particular, you could try making `nstart` bigger. This is *guaranteed* to make your results more stable, because the LDA function will just run over and over again with different random seeds and then return the best result. Unfortunately, increasing `nstart` to, say, 1000 will make the algorithm do 1000 times more work (cont'd)
Do this question and the answer refer solely to machine-learning?
I've just added the final part of my question, as I don't think it was clear - I want R to be SPSS for me for a moment! 

I'm aware it's bad practise to try to fit a foot into a glove this way, but I assume there must be a way to do standard statistical control without running iterations of models on residuals as described here: http://stats.stackexchange.com/questions/17336/how-exactly-does-one-control-for-other-variables
That's really helpful, thank you
@Bruder I actually went out and bought the book but found it lame in many regards.
Thanks, now it's more clear to me.
If this is discrete time, you must have a lot of ties, right? I'm under the impression that `coxph(ties="exact")`, in the standard `survival` package, makes the model "a conditional logistic model, and is appropriate when the times are a small set of discrete values". Would this not work for you? Is that b/c it wouldn't be using the `cloglog` link?
Would the down-voter care to give me some constructive guidance on how to improve my answer please?
`If I consider each city to be a n-dimensional vector with missing data turned to 0` The question is whether recoding missing into valid 0 is warranted. For binary (0 vs 1) data 0 means "absent" while missing means "absent or present - not known". No similarity measure itself can help you decide how to go about missing data.
@User603 Well, we can see that it happens for small $\\nu$ out as far as we care, but that doesn't prove it for all $\\nu$ (I can produce a number of series expansions that suggest it); I meant that one can make some arguments that it holds for all sufficiently large $\\nu$ (in fact whuber presents one in his comment).
Since apparently you (@phedonrousou) have repeated measures on the same subjects, this paper might prove useful: Myles & Cui, Using the Bland–Altman method to measure agreement with repeated measures - British Journal of Anaesthesia 309–11 (2007), doi:10.1093/bja/aem214
Say, I was just looking for this algorithm for a completely unrelated thing! Thanks.
If you have both industry type and access to the sea as independent variables then it is *not* a t-test, it is linear regression with dummy coded variables (aka ANOVA).
It appears this would force year-on-year comparisons to have equal means and variances, artificially eliminating most information about temporal change.
I can conceive that someone *might choose* to apply a more stringent significance level on interactions. Since the number of interactions and higher order interactions grow at a rapid rate (if there are enough factors), I can see some argument for it, such as if one were trying to make a similar number of type I errors at each order. I don't see how that somewhat plausible argument leads to an 'ought', though.
It's one of those judgement calls you have to make when model building. The last time I used an offset was modelling a Poisson process, that is, the response was the number of observed events in a time interval. As the time intervals were of the different lengths, I used $\\log(\\text{interval length})$ as the offset.
Gephi
http://gephi.org/users/quick-start/
Might help if you explain what kind of model you're trying to fit.
It is not a 1D optimization. But I have tried to simplify the issue as much as I can, as the actually question is quite complex. The answer I wanted to get was what you have given, around optimization and boundry solutions. So thank-you. I think I will re-write the function to by-pass the optim and just run an exhaustive search of parameters. I know this isn't the most elegant way to find the result, but it will give the answer I want and will shed some light on a different area of the problem that I think will be interesting.
Thank you for your reply Conor. I edited my original question and emphasized the 'noint' and 'with int' statements. Does anybody else have an answer to my question about when to use the BY and WITH command and what the exact difference is between the two?
As I believe William Gibson noted, the future is already here. It's just not very evenly distributed.
I believe that pdf doc you base yourself on is sure correct (and is quite mathematical). But I suspect it is for 2-class case only. The algorithm outlined by me ([extraction](http://stats.stackexchange.com/a/48859/3277), [classification](http://stats.stackexchange.com/a/31384/3277)) is general k-class LDA algorithm.
I have temporal dataset, I need to split it in epochs (of various lenght) and then I need learn one distinct model for every epochs. The score is the likelihood of the sequence of model in respect of the dataset partitioned in epochs. If I split good I get a good score, if I split bad I get a bad score. It's a possibility to use $\\exp(\\ln(score($new_state$))−\\ln(score($old_state$)))$ ? there is more good possibility?
Well, the model is of stellar structure. I have a code (Fortran) that, given parameters like the mass of the star and its chemical content, it solves a system of about 11 differential equations to produce a model, from which you can take observable properties like the brightness of the star. I'm starting with measurements of such things (e.g. brightness) and trying to get the best-fitting model parameters (e.g. mass). I'm using a MCMC to get the posterior distributions of the parameters, given a likelihood of the model matching the observations, within Gaussian errors. Does that help...?
Is there a reason not to go with a penalized regression approach (lasso, elasticnet, etc.)?
In the fMRI field, which is new to me, I understand this to be a standard analysis. I have time series and I want to know the dependence between them bc it relates one region to another.
Thanks for your feedback and I apologize for the delay. As for (3), I don't know how to answer that because they were both controlled by me when creating the test. I established the test such that x2 and x3 were held constant while x1 was changed. Then I held x1 and x3 constant while x2 was changed. Finally, I held x1 and x2 constant while x3 was changed. As I said, I'm really not particularly good with statistics, but what I'd really like to know is which of these variable holds the most influence over the result and I just don't know how to achieve that.
Gelman and Hill, is what I meant to say.
@Aniko It should be possible to adjust for the bias when early termination occurs.  We seem therefore to be discussing the naive use of a standard estimator, intended for fixed-size random samples, in conditionally terminated experiments, where such estimators do not have their desired properties. (+1, by the way.)
Thanks Mike, that is very helpful. Would you mind explaining why $cov(a_t,z_{t-1}) = 0$? Is it because a there can't be a covariance between $z_{t-1}$ and a future error term, $a_t$?
Also, I recommend you to take a look at this book, it's easy to understand and has some pretty good explanations in Bayesian prediction and in analysis via HDI: [link](http://www.amazon.com/Doing-Bayesian-Data-Analysis-Tutorial/dp/0123814855)
What is the total sample size and the distribution of $Y$?  Data splitting requires enormous sample sizes to achieve sufficient stability and precision.
By the way, I checked out Freedman. It is a joy to read, and is sufficiently mathematical for my purposes. Thank you for telling me about it.
I had more in mind the real distance in centimeters or inches. And then taking $\\log (1/d)$ assuming that $d$ is never 0.
I can reproduce these data *exactly,* assuming they are rounded to the nearest integer.  The formula is $y = 2528180 - \\exp(11.519 -0.386936x)$, $x=1,2,\\ldots,12$.  This formula forecasts the next value as $2527522$.
The problem with reckoning parametric approaches with the semi-parametric Cox Model is that this specific problem is actually related to a missing data. Despite that the author hasn't described how he's obtaining absolute risk predictions from a Cox model, given that we have such a risk prediction based on model parameters (and estimates of the baseline hazard function), the inclusion probability in the second phase of data collection depends on the original risk prediction, so missingness depends on observed variables, i.e. missing at random data.
I am in a exploratory phase where given two (and really many more) variables figure out the outcome. If the two variables are co-related then some of the variance is overlapping. I would like to know how much overlaps and how to weight the variables in an ending regression equation. (this my not be the right terminology so please bear with me.)
Do you really want to do an FMRI analysis with absolutely no chance of localization of the functional differences?
I have corrected the $sinh^{-1}$ instead of $sinh$.
You were right, you should buy a boat :-) I will create a reproducible example tomorrow. Basically, when the variable is ordered it gives the .L and .Q whereas when it is a factor variable just the comparisons with the reference category.
If you have data that's roughly normalish, I'd suggest a paired-t-test. If you don't, there are a variety of choices, but if the differences aren't too skew, I'd lean toward a permutation test.
The issue concerning comparing two beta distributions is successfully addressed at http://stats.stackexchange.com/questions/25290/what-is-the-probability-px-y-given-x-bea1-b1-and-y-bea2-b2-and-x.
In the book statistics for high dimensional data they said :" The empirical fact that often $S_0\\subset \\hat S$ 
is supported by theory. " Where can I found that theory?
@charles Thanks.I'll take a look at those packages. The way I came across with is using multinomial regression twice.First setting the last category as reference, by setting `ln[P(Class=2)/P(Class=3)]` equal to 0 I can find corresponding score for boundary between second and third category. Then by setting the second category as reference I can find boundary between the first and second category by setting `ln[P(Class=1)/P(Class=2)]` equal to 0.
@whuber - good point, I will edit the answer to reflect this
A reproducible example would be very helpful. The `predict()` method should work just fine in recent versions (>1.0) of lme4 ... you might find something useful in http://rpubs.com/bbolker/glmmchapter