tttjjj/elt_bench huggingface dataset:
@Chris: Let me see if I've got this right. You have time series data for $C_p(t)$ and $C_t(t)$ and the object of the exercise is to estimate the redistribution rate constant $\\frac{K_{trans}}{v_e}$ using that funky pharmacokinetic convolution integral (Eq. 4) as the systematic part of the model. If I've got this correct, do you have a model in mind for the error in the time series? Some sensible numbers for simulating data with which to test DE-MC would be helpful too.
I would suggest an older article by Bradley Efron and Carl Morris, [Stein's Paradox in Statistics (1977)](http://dx.doi.org/10.1038/scientificamerican0577-119) (an online [PDF is here](http://cs1.cs.nyu.edu/~roweis/csc2515-2006/readings/stein_sciam.pdf)). Not sure if it is intuitive, but it is a pretty gentle introduction (with real world examples) into the concept of shrinkage.
If you take the data every day, don't you know the day of the week?  And why would you suppose there is a binomial distribution?
@Srikant: The distribution of the product of n uniforms is the exponential of a (negated) Gamma(n) variate.
I have just implemented both the ideas. Thanks. I'd try and further dive into pca and networks approach.
Perfect, exactly what I needed. The paper here: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1950645/ threw me a bit with formula 1 (since it showed my density) but after reading your's I realized that they are summing only k-1 terms. Thanks again.
Categorizing continuous data (eg, age) is always a bad idea. Multiple regression & ANCOVA are ultimately the same thing, but sometimes software present them differently; it might be best to think of this as a basic multiple regression. A bigger issue, though, is that it sounds like you want to affirm the null, which is logically impossible. It may help you to read my answer here: [Why do statisticians say a non-significant result means "you can't reject the null", rather than accepting the null hypothesis?](http://stats.stackexchange.com/a/85914/7290)
Thanks for your reply. But, I still want to use calculus of variations.
Taking the derivative of the the pdf with respect to $c$ introduces the factor of $log(x)$ you are looking for: that's why digamma shows up.
+1 to this and to Greg Snow. I totally agree with this advice. @Nikolaus I think this looks "gaussian enough" to fit a mixture of gaussians distributions. You don’t want a perfect fit, just a way to check how many clusters there are. In this optic, constraining all components to share the same standard deviation can be a good idea (for the reasons explained by Anony-Mousse).
What do you want to use the chart for?
Perhaps your notation is confusing you, so let's write the decomposition as $P'\\Lambda D\\Lambda'P = LL'$.  Now isn't it obvious that $L = P'\\Lambda \\sqrt{D}$ where the square root is applied to the diagonal elements of $D$?
anyway, I'll give you the bounty for taking the effort.
@user1546029 I just hope I'm not doing your homework...
Please precise. What is the R package you use ? What are the output values you are talking about ? Are they the estimates of the parameters ? Why can they only be greater than 0 ?
I added the background of the problem if you're interested. Probably this will help getting a clue about what I search for.
What does your notation "w^(0)" refer to and what does it have to do with the Cholesky decomposition shown in the code snippet?
Folks, this is a question about the **path diagram tools**, not about R packages doing SEM. If the OP has an Mplus license, and is happy with the package, there will be little need for them to learn `OpenMx` or `lavaan`. These may be great packages *per se* if you start from ground zero, but unlearning Mplus and learning R/OpenMx is a huge time commitment that I personally would not be prepared to make, the most realistic setting for which could be getting tenured and going on a sabbatical to UCLA :).
Thanks! Your answer makes the solution very clear. It's essentially a broken stick regression, where the second basis function can only taken on two values (0 or 11), I think. Also, it will work cleanly with modifications I wanted to make but was worried about how to handle outside of the realm of a single model (random effects for repeated observations, a power transformation of the response to normalize the residuals).
@whuber The detection of a level shift is accomplishe by Intervention Detection (note this is not the same as Interevntion Modeling) Essentially it is sequence of trial baloons taht are evaluated based upon the effeciveness of the candidate structure. The best way to understand this is to see Tsay http://www.unc.edu/~jbhill/tsay.pdf.
For question 2): I've never heard of using the standard error of the estimates for model comparison.  What you want to do with the data? Variable selection?  Prediction?
@MichaelChernick Is this the [link](http://stats.stackexchange.com/questions/30075/multi-class-lda-vs-2-class-lda) you are looking for?
I had the exact same problem
What does the pattern of data look like?
@MikeWierzbicki: I think we need to be very careful, in particular with what we mean by *asymptotically unbiased*. There are at least two different concepts that often receive this name and it's important to distinguish them. Note that it is *not true* in general that a consistent estimator is asymptotically unbiased in the sense that $\\mathbb E T_n \\to \\theta$ even when the mean $\\theta_n = \\mathbb E T_n$ exists for all $n$. Many people call the convergence $\\mathbb E T_n \\to \\theta$ *unbiasedness in the limit* or *approximate unbiasedness*...(cont.)
@Orp Is the behaviour measured on a single variable or multiple variables?
It sounds like your weighting scheme was somewhat arbitrary: that would make it difficult for anyone to assess your procedure objectively. Have you done any testing of the accuracy of your classifier?
@whuber I was under the impression that this specific expression related to non-negative random variables, no?
@robin We don't have a policy that we can not ask question, before you have googled them. We want to be the resource that comes on top when you google this question :)
A meaningful control manipulation could be some general reinforcement (“You're doing good, we just need to do it again”). It would provide a check against general learning/practice effects and help decide if the specific feedback you provided is the active ingredient. Asking one group to complete two runs without any particular explanation in-between could be another potential control manipulation.
I like your way of thinking. You are asking: If a person's unobserved utility increases by 1.865 units with a donut option, then how much of increase in price creates enough dis-utility to offset that? and the answer is 3.57 or 357 dollars.  But then why have the authors said over 300, why not over 350? Specially since the cost differences is the main point here and they want to highlight the differences!
$\\hat{\\beta}_1$ is the estimated regression slope.  What does a confidence interval on the slope show?  Does it include 1?  Under the assumption $\\sigma^2_X = \\sigma^2_Y=\\sigma^2$, the variance of $\\frac{X+Y}{2}$ is about $0.98\\sigma^2$.  There is enough goofiness in the plot (e.g., the bifurcation in $Y$ measurement at $X \\approx 55$) that I'd say using $Y$ buys more heartache than improvement.
I think we should encourage posters to post links as part of the question if they have posted the same question at another SE site.
I think your frequency parameter is wrong here. I guess you want to add weekly seasonality, so you probably should change it to `ElecTimeSeries <- ts(Elec$Revenue, frequency=7)`. Just a side note, it is hard to help you with this without an actual data
:utdiscant Let me clarify what I mean by "given". If you assume that you have 1 endogenous series and 3 exogenous series then you can build an ARMAX model. To forecast this one endogenous series you might need future values of the exogenous series depending on the evidented lag structure and the length of the required forecast. The future values of the exogenous series would have to be delivered either as user-specified or the software might self-forecast these series using the appropriate ARIMA structure.If this is still not clear please either arrange for a chat room or contact me via email.
I do not think advising to build one's own generator is a sensible piece of advice. Existing generators have been tested and cross-tested, stick to them!
"Standard error" simply means "the standard deviation of the distribution of". The problem is that the distribution of $\\bar y$ has a standard deviation (i.e. the standard error), but if that's what you mean then the formula in your post is wrong. You may instead mean "standard deviation of the individual $y$'s", but if you assume Poisson, why would you do it that way?
If the model is correctly specified and if $Prob[Y=1 | X]$ is "transportable", i.e., $X$ is inclusive of all variables that might affect the incidence of $Y=1$ in new samples (i.e., you don't have a shift in the incidence of $Y=1$ in a new sample due to an uncontrolled/unspecified variable; this would also mess up any classifier), then the binary logistic model (and the other logistic variants) correctly models $Prob[Y=1 | X]$ in the long run.  An example where these restrictions apply is a cohort study whose subjects are not selected on unspecified X.  No prior is needed.
You need to know (or at least assume) a distribution.  Even then the uncertainty could be substantial especially if your distribution may have long tails. Try reading about the [German tank problem](http://en.wikipedia.org/wiki/German_tank_problem) for a simple example with a uniform distribution.
what type of object is mt4?  Is that the name of the dataframe you pasted in?  If you do str(mt4) what is the result?
thanks! The data point will enter a meta-analysis so there will not be any wording! I just wanted to check that it is ok to claim that the interventon group did better than the control group (but this was becuase the control group gained weight rather than the intervention group losing weight). I think from what you say it is fine to assert that the intervention group did better. thank you.
Michael, this question is so broad (and probably a duplicate), it's hard to say whether this (or **any** answer) answers the question at all. I wish you'd use your high rep status *to help us clean up low quality, possibly duplicate, questions like this by casting close votes(or explaining why you think the close votes are misguided)* rather than just posting a response that may or may not answer the question. Since you have the expertise to tell when a question is overly broad and you are on the site so frequently, your help could really be valuable.
What kind of thing or observation do these values represent?
'Eta' is a Greek letter, $\\eta$, not a statistical term. Have you really no more information about it, e.g. from the SPSS help files?
This is exactly what I pointed out in the question.
I think the the errors must be independent to be able to divide them by m.
By symmetrical, you mean the variance of $y$ is symmetrical around the mean of $y$, correct? In the generalized case where this is true, is the uncertainty of each parameter also symmetrical around the prediction of that parameter?  If that's true, then what you say makes perfect sense.
I see you're still working on this thing. I suggest you construct the joint distribution of $(X_1, X_2, \\mu_1, \\mu_2)$. Once you have (the mean vector and covariance matrix of) the full joint distribution in hand, calculating marginals and conditionals can be done straightforwardly using formulas in the Wikipedia article on the multivariate normal distribution.
@Readon Thanks for the background; the problem sounds interesting.  I tried downloading your data (twice) from the link you gave julien, but it seems to be in some sort of binary format which I'm not sure how to read.  If you'd export your data into a text file (or similar) I'd take a look at it.  I'm not sure what you mean by "gap  between the two basic distributions in the pdf", but if it means what it sounds like it means, then there might be something else you could do.
You don't test a correlation (however broadly or loosely defined) between two variables by comparing the frequency distributions.
This is a difficult question for this forum. Questions focused on how to use software are off-topic here, but this one would not go down well on Stack Overflow either as it is too open-ended. (I monitor all questions on Stata there.) Questions about specific statistical issues are welcome here, but there are too many things in which you are interested (two part models, bounded variables, skewness, regression) to allow an answer with reasonable scope. I'd recommend study of the advice in the Help center and more searching of the site before coming back with a specific question.
A low p-value implies that the two distributions are different.
Note that for large $n$, the asymptotic theory result was given here: http://stats.stackexchange.com/q/34418/5739
No, the null hypotheses are not equivalent (imagine a "X" shaped scatter plot: the mean difference will be around zero but the regression slope is 0, not 1). What I said is: If the individual differences are small, then the regression slope will be around 1. That's the only relation between the two approaches. They investigate different aspects of the relation between X and Y.
You may want to read my answer here: [ancova-and-its-disturbing-assumptions](http://stats.stackexchange.com/questions/44928//44959#44959). If you have any further questions, edit your Q to clarify them. I don't see that you have any problem; there is nothing wrong with a multiple regression model w/ just 1 IV, 1 covariate, & 1 interaction.
Hi BurningJade, welcome to CrossValidated. The `homework` tag says "A routine question from a textbook, course, or test used for a class or self-study. This community's policy is to "provide helpful hints."" -- since this is a routine question from a test being (presumably) used for self study, it counts as homework for the purposes of the tag; it affects the style of answer you get. See the brief paragraph on homework questions in the [faq](http://stats.stackexchange.com/faq).
The theoretical uncertainty is so small in this case that it can be ignored (for example it is much less than the precision of your experimental results).
This thesis is as implausible as it is illogical. As Stephen Stigler points out in his books, psychologists, economists, and most other social scientists did *not* adopt the physicists' methods for up to another century due to real doubts about their applicability and their interpretation. That is *prima facie* evidence that statistics is far more than a branch of physics. Other disciplines, ranging from engineering through biology, also employ physical methods and physical theories, but that does not make them branches of physics either--at least not in any meaningful or insightful way.
And it's best to think of and treat the linear and quadratic terms as a single unit.  In assessing the extent of a nonlinear relationship, you shouldn't incorporate one without the other, and if you run a sig. test you should check for the significance of their *joint* inclusion.
Actually a CLT *does* apply to the Cauchy, just not the usual one. The Cauchy is its own domain of attraction under means!
I'm not wanting to work out the skew of my data. I know it's skewed. I want to know if my data follows a skew-normal distribution.
Yes, I am checking also the silhouette plots and the best results are still with log-likelihood distance. (the IBM link you provided seem not to work, is there a mispelling maybe?)
@rolando2 I wasn't suggesting it was a property of a single variable (though I guess I can see how you might have taken it that way). I've addressed your comments in my answer above. I'll happily expand on them at greater length.
A standard scenario is that groups differ on the covariate. Thus, once you adjust for baseline differences between groups on the covariate there are no longer group differences. Things are more complex when it comes to interactions, but the same idea holds.
It would be interesting for the reader (and the OP) if you could provide details about the different R syntax (and the underlying tests) you gave, and especially how a Kruskal-Wallis test does compare to a log-linear model.
0-1 errors, i.e. FP error, FN error.
1. Most (all?) ML algorithms which do not explicitly require center/scaling as a preparation step will not be alleviated of any convergence issues by centering/scaling. 2. Normalized data leads to less interpretable model coefficients, since e.g. Pearson correlation is a unitless quantity and hence has no contextual importance. I suspect only PCA and related variance decomposition algorithms require some form of normalization.
@statskeptic I fyou saw my original post I apologize for my inital comments.  They were rightfully edited out by a moderator.  I think I misunderstood what you were trying to say.  I hope we answered your question well and relieved some of your skepticism.
@Lenwood http://docs.scipy.org/doc/scipy/reference/spatial.distance.html provides some common distance metrics. Each one has different properties so it is impossible to discuss all of them, but one example is Hamming distance which is usually used for string comparison. Another example is Pearson correlation distance which differs from Euclidean distance in that each vector is normalized, so [1,2,3,4] would be close to [10,20,30,40] in correlation distance but far in Euclidean distance.
Please read the `self-study` [guidelines](http://stats.stackexchange.com/tags/self-study/info) on answering homework questions. I think this goes far beyond helpful hints and strays into doing someone's homework for them, robbing them of the opportunity to actually learn the material.
Checking on amazon I found that Iacus's book looks like it is a nice introductory book with applications to statistical inference. I am not sure if it meets the OPs goal of providing good explanations of the numerical methods used to solve the equations. The first book recommended by beginner sounds like it covers the numerical methods and is highly regarded in the customer reviews on amazon.
In which case, that single mean value can be compared with the distribution of single mean values generated via resampling procedures such as permutation tests.
Well, I would expect that you will have a `rigorous definition of an outlier` when you'll be able to define `unreasonable amounts of subjectivity` objective manner ;-), Thanks
Thanks for the reference to Bloomfield's book. But could you please elucidate the difference between span=3 vs span c(3,3)? I read the help file, but I don't get it.
Try taking the natural logarithm of a number in (0,1).
Thanks .. now I understand
Could you clarify please, which three null hypotheses you mean, and what you mean by "the initial hypothesis"? When you say 'solving the problem', which problem do you mean?
Does the random slopes model in this case have a correlation between the random slope and random intercept, or in this framework is such an idea nonsensical?
This article provides a mathematical description how to test for power law distributions, as well as the R code. Clauset et al. "Power Law Distributions in Empirical Data."
It's similar to OLS, plotting the deviance residuals by sample and looking for any type of pattern would violate independence. http://www.r-bloggers.com/veterinary-epidemiologic-research-glm-evaluating-logistic-regression-models-part-3/
@sashkello Thanks for that. This can probably be closed as a duplicate, and somehow I missed that question entirely. One missing bit might be: If I train with `data_set1`, what do I consider the step performed by `LGOCV` cross-validation? From my reading I'm assuming 1) `caret` iterates through tuning parameters on `data_set1` and then 2) holds those params fixed and 3) creates a "sub model" using params from #1 for each `p = 0.8` sample of `data_set1` and tests predictions on the remaining 0.2 to gauge accuracy. Is that a reasonable summary?
The solution is given at http://stats.stackexchange.com/questions/37526/problems-with-groups-in-traminer
Can i ask you which model do you use the missing value imputation?
Here's an example using R: http://stats.stackexchange.com/questions/8918/is-there-a-way-to-optimize-regression-according-to-a-specific-criterion/8932#8932
I'm pretty sure the overlap coefficient is supposed to be restricted to the unit interval; this result is not.
Can you look at my new answer and see if that helps you to work it out? The facts I am referencing must be very well known, but I have never seen it being written out (except in my own notes), maybe somebody else have a reference.
Sorry, I've no idea what you mean by that. How could you even use "the same" estimator - $\\frac{\\sum(X_i)}{n}$ - in this case when you can't distinguish $X=1$ from $X=2$?
@whuber: (1) We can test the symmetry of a distribution around 0 by Wilcoxon sign rank test, based on its sample. This can be done, by letting $x_{1i} = 0, \\forall i$ in [wikipedia](http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test).  (2) How do people tell if a testing procedure is "valid" for a task?
(i) Did you mean to take the product of a pdf and a pmf there where you say P(X)*P(Y)? That doesn't seem to match your title which implies a product of random variables. (ii) You don't state what the bivariate distribution of $X$ and $Y$ is, only the margins.
Is your target variable $y \\in {0,1}$ or is it continuous?  If the former, then you could frame the model as $p(y=1) = \\Phi(f(x,a))$ instead of having the additive error term, and compare predicted with actual $y=0$ and $y=1$ to get the true and false positive rates, or compare to a baseline model where $p(y=1) = \\bar{y}$, or deviance, or several other alternatives.  If the latter, what's the distribution you're assuming for the residual?
@IMA: To be more specific my dependent variable is year-on-year change in population and the main regressors are year-on-year changes in other variables. Does the fact I am using changes get around these problems at all? (I am thinking it would as long as I can assume the city specific factors are constant).
Do I understand you correct: You have 17 input neurons, 30 hidden neurons and only one output neuron?
Thanks @B_Miner. I updated my answer with more answers :)
The ones in `plot(g1)` are probably a good starting point.
You can find a lot of information about this formula by searching our site on [standard binomial](http://stats.stackexchange.com/search?q=binomial+standard).  It also appears prominently in a [relevant Wikipedia article](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval).  None of these explain why $n$ has been replaced by $n-1$ in your formula, though; it may have been inspired by other variance estimators: see http://stats.stackexchange.com/questions/58230/degrees-of-freedom-for-standard-deviation-of-sample.
@Germaniawerks I'm saying that with intercept (2 parameters) there will be a better model fit than without intercept (1 parameter). One of us must be misreading something?
@Angelo Yes, the combinations could be associated with any other information. You should provide an example.
An idea: Run a regression tree and pick the leaf with the highest success probability. Relevant interactions are automatically chosen by the separation method.
Any guesses on what the "Chi Df" column is about?
You might find the answer to this question http://stats.stackexchange.com/questions/18937/adjusting-for-tilt-of-the-earth/19078#19078 relevant.
Right, 2000 is towards the bottom end of enough.  We are trying to estimate something that we can't see directly, so it takes a lot of data.