tttjjj/elt_bench huggingface dataset:
[tag:self-study] questions look for *guidance* instead of detailed answers. You could, e.g., skip from the second to the last line (omitting two mistakes) by noting the argument of $\\exp$ is a quadratic function of $\\theta$. Its first two coefficients can be read right off: the coefficient of $\\theta^2$ is evidently $-n/2$ from the sum plus $-t^2/2$ and the coefficient of $\\theta$ can be found by letting $\\theta=0$ in the derivative $\\sum_i(x_i-\\theta)-t^2\\theta$, giving $n\\mu$ (*not* $\\mu$). The constant coefficient (not involving $\\theta$) will be absorbed in the constant of proportionality.
@Scortchi the first one: I want to know the probability that the aggregated demand is below a given value: $P(X_1 + X_2 + X_3 < k)$
Thank you for your answer. I had previously looked at using multiple graphs and it seem that this is the way to go. I just wanted to check that there wasn't a better way that I had overlooked.
This is an amazing answer.  You win at the internet for this.
@Bernd Sorry, I don't.
Ultimately, the information in that document (which is very useful, thank you) needs to get incorporated in your question. But first, could you clarify what kind of `R` objects are `domain.of.choice_stats` and `objects.of.choice_stats`?  I suspect they might be matrices, whereas `cor.test` expects a pair of *vectors* and thereby mixes all your data together, producing a meaningless result.
http://r.789695.n4.nabble.com/Partial-R-square-in-multiple-linear-regression-td4632130.html
@user4272 - Indeed, as Dmitrij said, I do not know the $Z$. I actually do know the $I$'s for each observation. I clarified this in the question.
Can you clarify the "etc." for us?  If you are able to summarize your data in a table N by years (where N might be planned or unplanned exits), take a look at chi-square tests.
By the way, the problem with the indexing was because you were definition partials of $v_i$ w.r.t. $m$ and then for some internal summations in this definition also reused $i$, but in reality it should have been a different index, e.g., $j$.  I know what you meant, but it seemed ambiguous.
I'm clueless here. Shouldn't you be using "clogistic" from library(Epi)? Sorry if I'm missing something.
What does the last line of the question say? What does the first line of your output say? What does `?prop.test` say about continuity corrections?
It really depends on what you mean by "variability". Toy example: Which set has more variability: (1,2,3,4,5)  (1,1,1,1,5), (1,5,1,5,1) etc.
Could someone just have a small look? I'm really stuck since I don't know if the results are correct and thus my approach is correct?
Your first line is great, but it seems to go astray after that. The problem specifically notes that A and B are independent though, which seems to rule out the rest of your answer.
@winchester I have the feeling that analysis of variance would be more appropriate.
The `str` command, e.g. `str(neuralnetobject)`, is useful in these situations.
@Gavin Simpson: I hope you won't mind if I ask a related question. Why is ranking important? Is there some effect is it addressing here?
Thanks for the detailed reply Lucas!
Please don't cross-post your questions here and on math.stackexchange.
You appear to have mixed up type I and type II errors.  The probability that H0 is incorrectly rejected is the type I error, and it is set by choosing the level of significance of the test.  Are you interested in calculating the type II error, i.e., the probability of failing to reject a false null hypothesis?
@BabakP - Im not particularly well versed in statistics, but ill give articulating it a try - In the simplest terms in trying to find if there are reliable or predictable instances where dataA moves and it triggers a move in dataB
the answers to this question: http://quant.stackexchange.com/q/121/108 may be of some use.
I am probably not clear enough in my writing, but my questions are not regarding implementation issue. For example, in the part where I'm asking about creating the weight variable, I don't mean to ask what command in R can help me do that. I was simply wondering if by that I would be violating any of the assumptions of the random forest.
@whuber, I did not but will in the future, but read somewhere that Filtration,martingales, etc is the foundation for these types of models
(+1): A couple comments: (i) The case $A = 0$ is an interesting special case and has an intuitively satisfying interpretation that might be worth remarking on, and (ii) Note that classifying by choosing some $A \\neq 0$ is equivalent to ignoring the fitted intercept estimate and instead modifying it to optimize a different criterion conditional on the estimates of the remaining parameters.
This question is on topic at Stats--and I'm pretty sure it would get good answers there--but *because you requested that it be migrated,* we're sending it along to SO.  If the SO community closes it, we would welcome it back at Stats.
if we take first half of a bell curve, then we truncate the original distribution. So [truncated normal distribution](http://en.wikipedia.org/wiki/Truncated_normal_distribution) might be something you are after.
Hint: Writing the formula in the location-scale form $y = (1 + \\exp(-b(x - \\log(a)/b)))^{-1}$ shows that if $x_0$ is the solution when $\\log(a)=0$ and $b=1$, then $x_0 + \\log(a)/b$ is the general solution. Can you find $x_0$ such that $1/2 = y = 1/(1 + \\exp(-x_0))$?
I would treat the results as a sample of a population of an infinite number of runs, **but I would like to double check** which is why I posted the question. If the runs are treated as a population it affects the notation I use when discussing the formulas (big `N` for population size vs. little `n` for sample size) as well as the formula for variance (divided by `N` for population vs. divided by `n - 1` for sample).
Who claims that?  It would help to have a reference, because in the reference there's likely to be an explanation.
$\\mu$ is a *logarithm*: what units of measure do you suppose it has?
Could you explain how this solution would detect "local irregularities"? Presenting a worked example would be extremely helpful. (To be honest, I'm suggesting you do this because in carrying out such an exercise I believe you will discover that your suggestion is not effective for outlier detection. But I could be wrong...)
The inverse gamma conjugate prior is given for $\\lambda^k$, not for $\\lambda$. This is, you have to use a different parameterisation: $f(x\\vert \\lambda, k)=\\dfrac{k}{\\lambda}x^{k-1}\\exp\\left(-\\dfrac{x^k}{\\lambda}\\right)$. Please, have a look at page 20 of [this document](http://www.johndcook.com/CompendiumOfConjugatePriors.pdf) by [John D. Cook](http://stats.stackexchange.com/users/319/john-d-cook).
@GaBorgulya I added some more information to the question
@lakesh: if you are assuming perfect positive correlation, b will be always positive while for perfect negative correlation, it is always negative; this is because `sd(x)` and `sd(y)` are positive numbers(ignoring the case of zero situation here.
Hi Mike, thank you for your feedback. Temperature levels X1....X5 are specific degree values (i.e., heat stimuli = 42C, 44C, 46C, 48C, 50C; cold stimuli = 8C, 6C, 4C, 2C, 0C); however, they can also be conceptualized as stimuli ranging from "likely to cause minimal discomfort (i.e., 42C or 8C)" to "likely to cause maximum discomfort permissible by my research ethics board (i.e., 50C, 0C)". With this being the case do you think that a series of univariate repeated measures or a multivariate analysis is ok?<br/>
I too am am interested to hear suggestions from any mixed effects/multivariate gurus
There is a fundamental contradiction in any instruction to "group [items] together to form factors in a model (without reducing too much data)."  Grouping items into factors *is* data reduction.  It will by definition reduce the amount of information as compared to using all 35 opinion items. (Though there are often good reasons for doing so, as the factor analysis literature explains.) Otherwise, it'd be helpful if you'd state the topic and list a sample opinion item including its response options.
'12 participants chose 1' out of how many? Do all the 6 numbers refer to the same participants? Do you truly want the sum of differences, even if some are negative, and cancel out the positive differences?
Do you mean is 11.0% significantly different from 6.4%, 12.7% etc?
Exactly. And to argue that there aren't any, you would need to see the pre-treatment trends in the outcome for the treatment and control groups.
Hello, user36649, and welcome to the site!  In this case, there are several plausible ways the percentage could be calculated; we'd need to know what your professor (I assume this is for a class, and you really should add the "homework" or "self-study" tag as appropriate) has already taught you.  Perhaps you could expand upon the question to fill in the gaps for those of us who aren't familiar with the content of the course.
Which hypothesis do you really want to test? When you average many samples, you are throwing information away so that all that is left is a roughly normal distribution which is only described by $2$ parameters. It's quite possible that you could start with enough information to distinguish $X$ from $Y$, perhaps you can tell that $P(X=0) \\ne P(Y=0)$, but that you lose this ability by averaging samples.
It explicitly incorporates the estimation uncertainty in the procedure. Its main problem is the intervals are much too wide; given the assumptions more precise intervals are easily generated.
Are you by any chance being asked to find the *observed* Fisher information?
So where I have mentioned $p$ values in my text, not the formula, is that correct or should that be $z$ values aswell?
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
For the 19 sample subjects there is a strong negative relation between the slope and the average of the 5 responses. If that holds in the full data set then you might want to reconsider the meaning of slope-based groups.
It is unfortunate that nobody could chime in with a package where this is easy to do.  However, it turned out that adding the appropriate functions to lme4 wasn't as difficult as I had feared.  Now the only challenge that remains is to find the "correct" link function (see: http://stats.stackexchange.com/questions/1583/appropriate-link-function-for-2afc-data).  So I'll accept ars' answer, but someone may want to ask this again in the future since next time the answer may be different.
I haven't tried anything so far, as I don't know how to proceed. In stata I can do   poisson y x , irr but in R all I know is glm(y~x,family=poisson(link = "log")) which doesn't give IRRs and their standard errors etc
+1 for an encouraging and helpful answer.
(3) what does ACF look like within lag(q-p)? What does PACF look like within lag(p-q)?
Just for your info: there is a number of related questions you might want to check. Search `lda decision boundary`.
Mybe you have a answers with hv block cross validation for autocorrelated data
Welcome to the site, Antonio! Given that another respondent has mentioned Gujarati and Porter in a positive light, it would be nice to have more specifics on your criticisms of it relative to the previous edition. :-)
So you asked randomly picked persons to tick at least one option? If yes, and if the sample size is not too small, your suggestion is quite ok. Usually, Fisher's exact tests are performed instead of two-sample t-tests in such a setting.
I calculated the pvalues using Heidelberger and Welch as well and they differ quite substantially from the pvalues I calculated from the z-score from Geweke's diagnostic. Also, I see that if the autocorrelation time is very low (<100), the p-values are very high. I must still be missing something...
Cristina, can you successfully do a *linear regression* on your data on the machine you are using? How long does it take? How long do, say, 10 or 20 repeat iterations take to run? This should give you a crude bound on what to expect for *logistic* regression. Much more than that and either your features are forcing a very difficult or unstable fit or you have an implementation bottleneck. Cheers.
"this would reflect we know nothing about parameter $p$" — it is a uniform prior, but it's not an uninformative prior.
So much homework, so little time....
That seems rather a general question - would you like to take a look at http://en.wikipedia.org/wiki/Latent_variable and see if that helps, or if not, which bits you don't follow?
That's perfect ! I was hoping for a pointer but that's a really good answer.  Thanks.
Sorry - I messed up when I copy-pasted the code. Now it is complete. S is the variance-covariance matrix of the residuals. You were on the right track (with the `vcv` function) -- but you need to get the correlation matrix and then use $\\hat{\\sigma}^2$ to turn this into the var-cov matrix.
@Silverfish Thank you for pointing that out.  I have double-checked the analysis, fixed the exposition (you were correct on all counts), and cleaned up the formatting a little.
Are the response variables the same in both cases (eg, both weights)? Are the covariates the same (eg, both weights)? Is this two different groups (eg, college basketball & football players) or two different studies? The answers to these questions would help people answer. Also, I notice that you've asked 9 questions & not accepted a single answer yet. You may want to click the 'check' mark next to answers that you've found helpful in your previous questions; it's a nice thank you to people who've helped you, and encourages people to keep helping you.
And can you provide the three formulas you are referring to (Entropy, Deviance, Impurity measure)? There are more then one.
As I understand the OP, he is not using one prior and looking at the results and deciding that another prior will give him better results. He is using an artificial prior to see that his model is properly coded, then switching to his actual prior for analysis. Perhaps if he also used synthetic data in his first step, then switched to both his actual prior and his actual data in the second step, it would be more acceptable?
What about replacing the variable "current age" by the follow-up time (=current age minus baseline age)? This helps to prevent interpretation problems due to multicollinearity and it fits better to the original question. Let's say if the effect on mean response (higher = better) of "baseline age" is 1 per year and of "follow-up" 2 per year, then I don't even need an interaction term to see that late entry is bad.
As long as the outcome is univariate/Bernoulli then sure.  Just pass a matrix or data-frame to the X argument with each column representing the predictions from a different model.  So matching your above example: `colAUC(X=cbind(x1,x2),y=y,plotROC=TRUE)`
I cannot imagine doing this being possible.
Maybe.  Non-parametric tests almost always compare *medians* (or other percentiles) rather than *means* and therefore really address a slightly different question.  But this does not seem like a helpful reply to the current question, which asks specifically (and only) about t-testing the logs of the data.
If the volume remains constant, then for an [ideal gas](http://en.wikipedia.org/wiki/Ideal_gas_law), the product of the temperature (measured from absolute zero, e.g. in Kelvin) and the pressure should be constant.
My question is sort of awkward, but I want to know how to construct 'simultaneous confidence intervals' if possible.
I am working on text classification and I am looking for a result which gives important points from the article, something like that.
This is called [linear interpolation](http://en.wikipedia.org/wiki/Linear_interpolation).
Please, explain the TLA (three letter abreviations). Also, is there some statistical content to this question or is it simply a "how do I do this in R" question? If the former, please improve the question by giving more details, explaining what you tried so far and providing a reproducible example. If the latter, improve the question by explaining what you tried so far and providing a reproducible example and the code and flag this for migration.
I have, but my issue is that I want to compute the correlations among subgroups of one variable. So when I create new variables to separate the subgroups, I encounter problems with the missing values.
Is there a book where I can find this sort of thing?
Actually I was more stupid than you image. I had thought BinSeg is Segment Neighbourhoods so it is exact... However, it seems in the example case in the post Binary Segmentation is superior to PELT in my test.
@Michelle I concur with Peter: if you are summarizing the parameters of the curve, then you get one parameter estimate from each curve, and that should be fine.
Andris, 

Thanks so much. I appreciate the help.

This was my first post, Is there a way for me to mark it.  I don't have enough karma for an up vote.
But if you knew the actual score for those samples, (variance 0), then you would have $\\hat{\\sigma}^2=0$
if they are normalized, they are the same. but think of what happen when you make change of units...
Added extra details in my post.
Related question here, [Displaying fine variation across multiple orders of magnitude](http://stats.stackexchange.com/q/76772/1036). Use a log scale is the simplest answer.
@MPel, being more conservative than Bonferroni is surprising, it is generally considered to be one of the more conservative approaches.  How does it compare if you only use the standard correlations?
I like pictures, and instinctively feel there's a great idea lurking here--I *love* the idea--, but I don't understand these particular ones.  Explanations would be helpful.  One thing that stops me in my tracks is the thought of trying to extend the integral of $(1-F(t))dt$ to $-\\infty$: it has to diverge.
It might help if you clarified the context. It seems like you're talking about a continuous predictor in logistic regression (as I've assumed in the answer below) but more information would be helpful. I have the same comment regarding your interaction question - http://stats.stackexchange.com/questions/22036/interaction-terms-interpretation
Sounds like a problem of complete or quasicomplete separation.
Thank you very for all your advises. It works very well!
A good question. As noted by the answerers, it amounts to the question how to EFA-analyze dataset with missing values (with MCAR pattern, if you give completely rundom subset of items to every respondent). You could impute. But any sort of imputation is warranted if % of missingness is not too high (usually, they say, no more than 20% of data cells). That means that you should give at least 80% of all the items to each respondent.
Sorry I just started learning simple linear regression and I am not very familiar with the matrix form notation. Could you tell me what it would be in simple linear regression?
Nice one too. Thanks ars!
The devil lies in defining things like your phrase "*to get a good idea of*"
Thanks for your answer and reference. I will check differents models and do cross validation. Bayesian is a good idea.
Very useful, very helpful.
Casino dice have their dots filled in with coloured material with the same density as the dice, and then its all sanded flush to a zillionth of an inch. Source: wikipedia
Just plug these 3 principal components as regressors into the model and compute their regression coefficients. These are either interpreted directly as effects of the principal components, or by 'inversion' of the PCA, as effects of the original variables.
@Marco. Can you please make edits to your original question? This will make it easier to answer similar questions in the future. It will also help me comprehend your design and answer it correctly. Thanks in advance.
Rather than complicate things with two separate decompositions, I would reduce the t.window to allow a more sudden change if you think the default value is smoothing it out.
@Selden: ok, did not order. now look if that's better. Generally it helps when the original poster provides code.
agree with etarion, there's barely anything related to programming here. On a side note, I have no idea about the question, sorry. :(
To compute similarity, I need to count the number of times two items were rated together by the same person.  That count becomes my numerator.  The formula I'm using to calculate similarity between items A & B is 

    AB / A + B - AB

Where AB is the number of times items A and B were rated by the same person, and A and B are the number of times each item was rated overall.

In order to perform that count, I need to generate all of the itemA,itemB pairs in my mapper, and then sum the counts in my reducer.

The items are rated in a binary fashion-if a (userid,itemid) exists, then 1.
"*because normal QQ plot were normally distributed for this model?*" ... I might say "QQ plot of residuals suggests that the assumption of normality is reasonable" or "residuals appear reasonably close to normal". If your audience expects hypothesis tests, you might still quote one (but that doesn't alter the fact that they're not particularly helpful). "*The problem with the dataset is that in the histogram of the DV*" ... there's no assumption about the distribution of the unconditional DV or any of the IVs.
Also, check the treatment of missing values. In your SPSS code, you delete them pairwise. In you R code - ...?
I agree that the estimate of variance for the total population is what you said.  But we're trying to estimate variance for the 24 samples.
I'd appreciate answers to the first three questions.
yes, this is very, very helpful, thank you!
By linearly separable, we mean the features to be linearly separable or the class?Another question, which is not based on this and it shall be grateful if you can answer which will spare another question which is - In learning or training, using input-output pairs, what exactly are we learning?IS network learning the inputs or the outputs?
+1 for the R code illustrating fast solution of the problem
I based my answer on your original question which made no mention of time series and the idea that this would be a multivariate time series problem makes me wonder how just saying manipulating data and smoothing would give anyone a clue as to what you are taking about.  Certainly there are smoothing techniques that can be applied in the frequency domain.  But maybe you should think about applying multivariate ARIMA type models. But please realize that fitting models and smoothing is not manipulating data!
@Firefeather - Yes I do. I'd like to know what's strictly right versus what's pragmatic to do - maybe modeling these as Gaussians isn't right, so adding or multiplying is the theoretical right answer but not practical.
Kind of the same as with 2x2 tables. Under the assumption of independence, these data are highly unlikely. If you want to report on the magnitude of the deviation from independence, you could take a look at pearson residuals.
+1 Wow, nice list.
Thank you! I will check both links you provided as soon as I find some time and get back to you.
Is there a name for what I am trying to do, in case I need to find out how to do it for another distribution? Is it "confidence interval," or is that something else?
Hi JC, Thanks for your reply. I've taken the approach of transforming both my dependent and independent variables for consistency, but I've read that it's only the DV that really needs transforming for normality compared to its IVs.
Thank you for your solution Vincent. Much appreciated! Will try this now.
Just to make it clear. You want to regress LN(Weights) on X, right? I don't see why it shouldn't work. LS estimation is not restricted to X unequal to zero. I even simulated some data in my excel with the first value for `x=0`. Then I clicked on the plotted line, then right click and added a linear trend line. Maybe it is not working because some of your weights are less or equal to zero.