tttjjj/elt_bench huggingface dataset:
@whuber, Agree. But the question links to my reply where I gave the concrete SPSS code, the model. Adhesh is expected to have used _that_ code, by the sound of his question.
First of all, thank you very much everybody for your quick answers! As it's "only" my bachelor thesis and my real estate professor's work regularly is more qualitative than quantitative (no econometrics), I want to avoid to complicate it for him by logistic transformation. I already did the regression on Y_3 with all the predictors, but I wondered if you could "see" the probably significant predictors before running the regression. As this seems definitely too much for the purpose of my analysis, I will do what you said: Y_3~X_1+... and check for the magnitude and the significances. Thank you!
It may be better to either show the _actual_ code you ran, to avoid these kinds of confusions (all too often, an assumption during the 'translation' turns out to be the real culprit). Either way: I cannot debug this since I can't run it, so your only option is to use the traceback, or do actual debugging.
as Michal J. Figurski says you need normally distributed data. Have a look at the Kruskal-Wallis test!
@Rob Caricature? I think this is what makes statistics beautiful! You assume all those gaussians and linearities and it just works -- and there is a reason for it which is called Taylor expansion. World is hell of a complex, but in linear approx. (which is often ninety-something% of complexity) embarrassingly trivial. ML (and nonparametric statistics) comes in in these few per cent of situations where some more subtle approach is needed. This is just no free lunch -- if you want theorems, you need assumptions; if you don't want assumptions, you need approximate methods.
Interesting. So there is no way to validate your results?
I think your question is still of great interest. Is there any "out there" who could answer it? The question: "My question is this, when using the spec.pgram() function in R, a confidence interval is calculated and plotted as a blue line -- what is the correct interpretation of this interval, and how can we extract what its numerical values are?"
+1 what a great answer.
You might look into the nonparametric confidence interval and estimator for the difference of location parameters based on the [Hodges-Lehmann estimator](http://en.wikipedia.org/wiki/Hodges%E2%80%93Lehmann_estimator). As explained in the [help page](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/wilcox.test.html) for R's `wilcox.test()` (under `Details`), this is closely related to the difference in medians, but not quite the same.
PCA is not invariant towards different scalings of the predictor variables. So you could try to perform PCA on the correlation instead of covariance matrix. But not sure it works because the magnitude difference is for you for one of the samples and not for some of the variables.
You might be interested in http://en.wikipedia.org/wiki/Ecological_fallacy, which applies here.
(ctd) ... However, the same test can also be done as an F-test, by comparing nested models, one with $\\theta$ in it, the other without.
I am not sure I understand. Doesn't the intercept refer to when all the predictors = 0, so when the user weren't exposed to any of the channels.
See this related question from math.stackexchange.com, which covers the case when d(x,x')=norm(x-x'), but not the general metric case.  http://math.stackexchange.com/questions/248976/how-to-prove-that-exponential-kernel-is-positive-definite
I extended my answer.
I like the first example you give. That will certainly get the students talking ;)
Correlations, eh?
I've tried using intra in inter cluster distance, but couldn't think of something usefull for a cluster with one point. Also I don't have a center point. I only have distances between points.
Yes, the data were scaled.
Sorry that you don't like my tone.  If you want to know whether you can reasonably assume that the two distributions are the same, then the KS will mislead you, because it tests the null hypothesis that the two distributions are the same.
What concrete questions do you want to answer?
An important point made in the threads I referenced in a comment--and I think this is the crux of the entire issue--is that all these methods are *arbitrary*. There exist objective methods but they require the use of information not inherent in the data. That's what makes this a problem of *valuation* rather than statistics.
Certainly, $X_1$ and $\\bar X_1$ do not have the same correlation with the other variables. For the core of the question, I never heard a term like prevalence in simulation...
The answers you got are fine;  just suggesting that this sort of question may get a lot more views in SO.
I'm not sure what you exactly your question is. Do you want to know which is the preferred performance measure from the list you provided? Is it how to combine the predictions of individual leafs into one performance measure? In general, the purpose of any validation procedure is to estimate the expected prediction error of your model, cross-validation is one way of doing that.
You might want to convert the data frame to "wide format". Then, note that the surv() function can take 3 arguments. The 1st argument is the starting time for the interval, the 2nd is the ending time of the interval, and the last one indicates whether the observation is an event or not. I cannot provide more details right now...
@ Sarah Thomas. The truth is that I did not use print function.Thanks for pointing it out. However, my professor told me that lme function should be used here. I will try with print and I will let you know.
How does WMW know if the variances are the same or different? It ignores variances. A deeper problem is that you say absolutely nothing about your sample sizes and little about the character or strength of non-normality in your data, so we don't have enough information to advise well. Tests being significant (at conventional levels) is not much information. Unimportant deviations from normality (e.g.) will often show up on (e.g.) a Shapiro-Wilk test whenever the sample size is large enough for them not to be a problem. But using Welch sounds a good idea.
@jbowman I'm not sure if I entirely understand your question.  I am ultimately trying to test if there is a difference between the two groups (e.g. does one group tend to have larger absolute values than the other group?)
Oh, OK! Well... if you did not find it, R is an open-source statistics environment and programming language. You can download it at http://www.r-project.org
@NeilG I'm not sure I understand your concern, although I'd be happy to make a correction if one is warranted.  From your other comments in this thread, it seems like you think the Jeffreys prior is the last word on what's uninformative.  That's not the case. See e.g. [here](http://andrewgelman.com/wp-content/uploads/2012/07/timeseries06272012.pdf). Especially given that User13&c was just looking for intuition and that we both agree that Gamma(0,0) won't actually work for their context, I'm not sure what the problem is.
By Fisher's method I assume you mean his method for combining p-values. You can't use that on significant/not significant results because you need the exact p-values (i.e. you need significance test results, not hypothesis test outcomes). There are many different methods to combine p-values, for example, see Whitlock (J. EVOL. BIOL. 18 (2005) 1368–1373). If you don't understand the distinction between hypothesis testing and significance tests (and many don't) then see my paper http://www.ncbi.nlm.nih.gov/pubmed/22394284 for an approachable explanation.
You are right. Use the censor information in the ML estimation.
@gung, No, no typo. $SCP_{ZZ}$ is correct (and is actually $SS_Z$)
I was trying to use kml, but an error occured. Can you help me what 'contains trajectories to cluster' means which is the first variable of procedure kml() ? I want to cluster variable units_sold by time, so i tried with t(units_sold) but it do not work
Substracting the mean and deviding by the standard deviation (centering and standardizing).
The Stata code looks rather inefficient, even accounting for it being four versions old. These days, all this machinery is bundled in `margins` and `marginsplot`, and they produce results that are quite different from what's published in the paper. Approaching this by simulation sounded a bit odd to me. This looks more like a job for Stata's `nlcom`. Ah well... it's not my paper that we are looking at here :)
@Arun - I recommend splitting this into smaller questions that illustrate the specific problems you are encountering. For instance, say you don't know how to get your four data files into R into an appropriate format. Ask a question about that - or better yet, check out this question or one of the many variants: http://stackoverflow.com/questions/5758084/loop-in-r-loading-files. As for the appropriate statistical test(s) to run, crossvalidated.com may be a better venue for those discussions. At a minimum, update your question with a reproducible set of data so others can offer tangible advice.
@ttnphns It seems to be this question is primarily conceptual: although it is framed in terms of using SPSS, the problem arises because the OP needs to better understand how to frame statistical problems and use statistical software in general. Ted: if you want this question to stand, please edit it to be more specific about what you have tried and what you are attempting to do ("compare" is too vague to be operational).
The title says "intersection", while a union symbol is used in the text. Which one do you want?
(+1) The color goes a long way in showing what is happening.
It would mean adding $2^{30}$ explanatory variables, which is approximately one bilion ($10^9$) variables. This is to all intends and purposes impossible
Each of my audio track would be just one instrument playing
You are drawing subsamples of size $6$ from an approximately uniform distribution.  For a truly uniform distribution the ratio is $10\\sqrt{3}/7\\approx 2.474$.  Indeed, if you were to use that factor in your simulation you would obtain `mean(R)/2.474` equal to $2887.6$, very close to `sd(x)`.
So, in essence, I think neither mean or median are good. Showing (as Amazon does) a bargraph with the different votes is the best option. Also, it is interesting to point out that online 1-5 ratings are not always so fair... http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html
I guess I'm a little confused by what you're seeking so I'll do my best to use the example you've posed. For a fixed value, let's say $y$, we wish to estimate $F(y)$. Then we take the $N=1000$ samples and get some value $\\hat{q}$. Now take $M$ replicants of the original sample and you repeat the same computation yielding $M$ values of $\\hat{q}^*$, giving a CI on $\\hat{q}$. The key point is that the original sample can get you the statistic and the bootstrap gives the CI. Perhaps the real question is what you want to do with the empirical distribution.
From http://en.wikipedia.org/wiki/Random_walk#Gaussian_random_walk we have this formula, which seems analogous to the situation I described, but for two players. "If a and b are positive integers, then the expected number of steps until a one-dimensional simple random walk starting at 0 first hits b or −a is ab. The probability that this walk will hit b before −a is a / (a + b), which can be derived from the fact that simple random walk is a martingale."
@gung: No, I don't believe there is anything precluding us from the typical $F$-test. The test does not depend on the presence of an intercept, and, indeed, I believe the $F$-statistic will work out in this case to be $$F = (n-2) \\left(\\frac{\\|\\y - \\ytilde\\|_2^2}{\\|\\y - \\yhat\\|_2^2} - 1 \\right)$$. This gives us a bit of quantitative information in that, if indeed $R_0^2 > R_1^2$, then we know that $$F < (n-2) \\frac{\\bar y^2}{n^{-1} \\|\\y - \\bar y \\one\\|_2^2} \\>,$$ assuming, of course, I've done the algebra correctly.
Welcome to CrossValidated, @jbruck. Please post your question as a separate question, instead of an answer. You can do this by clicking the "ASK QUESTION" button. Also see the about page here: http://stats.stackexchange.com/about.
Fixed effects:
                         			Estimate 	Std. Error 	z value 		Pr(>|z|)    
(Intercept)                 		3.19504    	0.90446   	3.533 		.000412 ***
Treatmentshiny_non-shiny	0.02617    	1.26964   	0.021 		.983558    
Trial                          		-1.53880    	0.36705  	-4.192 		2.76e-05 ***
Treatment:Trial  		0.16909    	0.49501   	0.342 		.732655    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Correlation of Fixed Effects:
            (Intr) Trtm_- Trial 
Trtmntshn_- -0.712              
Trial       -0.895  0.638       
Trtmnts_-:T  0.664 -0.896 -0.742
see updated answer.
@user13467 I added error handling. See update.
It looks like this comes down to the question of practical vs statistical significance.  Visually you are not seeing much difference between the groups, but the tests are significant.  You need to ask yourself if they are close enough for what you want to do and worry less about tests of very narrow hypotheses.  Consider the 2 distributions, one is uniform from 0 to 0.99, the other is uniform from 0.01 to 1, are they different, the tests can say yes given enough data, but for most practical considerations the differences won't matter.
Thanks. This may give some insight into the problem I am working on.
gmm is for gaussian mixture model
OK, I will take a look at this. I'll leave the question open, though, in case anyone else has something to add.
As a routine textbook type question, this counts as `self-study`. Please add the `self-study` tag and review its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info). With that in mind, what have you tried, and where does your difficulty lie?
A simple start could be to run a logistic regression for each intervention (physical restraint yes = 1 vs no = 0) using gender, age group etc. as predictors. More complicated techniques include baseline multinomial regression or multivariate logistic regression.
As I understand it, traditional methods in machine learning don't consider correlation between rows of features in the training set. However, that is the most salient feature in time series data. Obviously, you could omit it and see how well your algorithm does, but in some application this is not feasible.
Are there multiple loans per customer?  If there's only one loan per customer, including factor(customer) in the model will explain all of the variation (and I presume that's not what you want to do).
How to get expression for CDF[DirichletDistribution[{1, 3, 2}]] ?
I expect that would come from an examination of the data based on the actual results
My answer at http://stats.stackexchange.com/a/46508 addresses this question beginning at the section "Multiple regression can be obtained by sequential matching." The short answer is that you are on to something, but at each step you have to regress *all* the variables (independent and independent) on the current variable and retain those residuals for the next step.
What you hint at I suspect as well, in that it is likely the random effects estimation that is causing differences between the two procedures (as opposed to the link function). The fact that it is different sets of data is immaterial (since you don't provide the data for people to examine and estimate the models themselves anyway). I would suggest you stick with one question, and when that is answered if it doesn't apply here, or doesn't satisfactorily answer this question, then ask a second one.
(1) If I were performing, say, the KPSS test, I'd allow for trend under the null unless I was sure that there wasn't any, thus allowing me to interpret a high value of the test statistic as evidence for the presence of a unit root (given some other assumptions). (2) What model? If you're suspecting a unit root an OLS fit isn't much use, as a high t-statistic could indicate that as much as it could trend.
Have you tried princomp function?
One option would be to fit a model where the returns of one series are explained by the returns of another and test for a structural break in the model.  It's not clear from your post whether you are actually concerned with returns or price levels however.  What you may be asking about is cointegration, but it's not clear from your post.
Then why do you have to measure with Machine 4? Why not measure it all with Machine 2? I think I'm missing something in your setup
No, the answer that supposedly used Bayes Rule is incorrect. I'm not sure why you are confused, MC's formula above is correct and cannot be used to get any result, that's what his and Whuber's answers to the question explained!
Although I am happy that you got an answer, this kind of question really doesn't fit here as it pivots on a particular software implementation. Please see the help center for advice on software-related questions.
The point is that I am interested to learn the probability for certain event to occur. For my training examples ${x_i,y_i)}$ ($x_i$ being a vector and $y_i$ a scalar) the probability that a given vector $x$ will produce an output $y$ or not. the number of times the output ($y$) is one is the number of observations for a given number of trials (say $n$). So I am interested in the probability that for a certain vector $x$, what is the ration $ \\sum(y)/n$, meaning, what is the probability that, that vector would produce an output of $y=1$ for a future vector $x$.
Ok. So this question's taken from a past exam paper for Bayesian Networks. I was given that matrix because there is a child node D with two parents E and F. However, at some point the question specifies that the link between E and D must be removed since E and D are independent. And the question asks to deduce P(D|F) from P(D|E&F). Therefore, I assume that the question must have an answer but you said that we cannot conclude anything. Thanks for answering tho!
Good answer--but it (and the original question) belong on a different site.
@Alexis, but doesn't Shannons's entropy range between 0 and 1?
Perhaps you should edit your question to ask about that. Can you say more about what you need to know to understand / trust it?
How often do you have the entire population of interest? How often is it even possible?
@mnel The function `fractions` will return the fractions as strings. I suppose its easier to use integers.
@aL3 Normality of the data is irrelevant: normality of the *residuals* is what matters in ANOVA.
1nn: what is then the continuous score you vary over the ROC? Application: the problem you want to solve, not the software.
@ariddell i agree, but it is commonly used in introductory NLP examples, and its large enough to be useful in learning but small enough to be analysed on a good laptop.
Can you expand on this a little? Perhaps can you derive why $E[Y \\mid X]= E[E[Y \\mid X,Z] \\mid X]$? Or better yet as Woolridge has: $E(u_{t}u_{s}|{\\bf x_{t}},{\\bf x_{s}})=E[E(u_{t}u_{s}|{\\bf x_{t}},{\\bf x_{s}},u_{s}){\\bf x_{t}},{\\bf x_{s}}]$
I must admit that I don't completely understand the question, but I am pretty sure that the suggestion of @user1121352 is incorrect. Units of variance are squares of units of the original data, and so variance does not scale linearly with data. Standard deviation, on the other hand does.  As I wrote, I do not completely understand the question, but in general, if you scale your data by a factor, standard deviation will scale by the same factor.
I find this disappointing -- I was hoping for more correlation between my prediction and actual observed values, similar to other forms of regression (I'm thinking least sq. for example). Do you think using the expected lifetime, instead of the median, would produce desired results?
not really : the slack would just go to infinity. hence my trouble and dismay...
All in all, probability distribution of parameter expresses our knowledge (as obtained by data alone, or by other means) about that parameter. Sense of a true model is not lost - you simple do not know with complete certainty which one of possible models is the true one. Therefore employ a measure (probability distribution) to describe available knowledge about the true model.
A good source of information to clarify differences between frequentist and Bayesian approaches might be this link
http://stats.stackexchange.com/questions/31867/bayesian-vs-frequentist-interpretations-of-probability
i have added an example data frame on my original question
I think the Lasso is popular because it effectively re-casts the model selection problem from one of hypothesis testing to one of parameter estimation.
@itsols  Technically, you should always specify which type of deviation statistic you are calculating for the data set -- the word deviation on its own should refer to the deviation of a single datapoint from the mean (in the way Kasper uses it in the answer).
.. handling multiple comparisons?
Could you be more specific about what you checked? For example, did you check if some variables should be recoded (e.g. items with a negative formulation)?
Do you know of a specific cluster analysis which allows me to specify how many clusters should exist? For example, what if I wanted EXACTLY 5 clusters? 10 clusters?
Firstly, you should probably use AICc, not pure AIC. That being said, it sounds as if that the information criterion is telling you that the additional effect does not help the likelihood as much as the additional parameter penalizes it. In other words, the additional variable isn't worth the parameter risk.
... or the question is suitably taylored to each site. Simply copying and pasting questions *simultaneously* at multiple sites is similar to spam (which is what the OP has done in this instance). I will bring up a meta post *on our sites meta* to discuss this when I get a chance.
I have already used an alternative methodology to answer this question. Beta is one of the important determinants for general large falls over the entire event window.
(+1) That is an awesome response, which demonstrates that a simple question about R data manipulation can lead to a nice discussion about problem reformulation and the choice of coding strategy. Moreover this solution has good timing, compared to [more basic](https://gist.github.com/2708618) approaches.
We do not "totally disregard" that the coefficient is jointly normally distributed with the others. Just look at how the variance of each coefficient is obtained to see that. And, there are plenty of tests that take into account multiple coefficients. The standard $F$-test is one, [Scheffe's method](http://en.wikipedia.org/wiki/Scheff%C3%A9%27s_method) is another, Wilks' generalized variance and ellipsoids of concentration yet another. It comes down to what statistical question you are interested in.
@JVM Can you explain how the getmeans function works, and why you didn't just use the mean or colMeans functions?
If some variables have a large variance and some small, PCA (maximizing variance) will load on the large variances. For example if you change one variable from km to cm (increasing its variance), it may go from having little impact to dominating the first principle component. If you want your PCA to be independent of such rescaling, standardizing the variables will do that. On the other hand, if the specific scale of your variables matters (in that you want your PCA to be in that scale), maybe you don't want to standardize.
These are two separate questions. The answer to the second is [yes](http://en.wikipedia.org/wiki/Principal_component_analysis#Relation_between_PCA_and_K-means_clustering). The first one is too vague. First what is the magnitude of the vector? Also what kind of relationship you are looking for? Principal eigenvectors in PCA are the eigenvectors of either covariance or correlation matrix of original variables, so they are clearly related to correlations.
To reliably estimate a coefficient takes a deal more than a couple of points per coefficient. You're talking about estimating 5050 parameters (plus an intercept and $\\sigma$). Do you have enough data for all that?