tttjjj/elt_bench huggingface dataset:
Thanks for reply.I have seen that when the number of predictors(m) is much larger than the number of observations(n) in high dimensional data then lars gets at most n steps, but I am not sure how I can define the exact number of steps in order to have the saturated fit. Also is it possible to give me a hint how I can define the parameter lambda for lasso in order to have the saturated fit?(any document that explain it). thanks in advance
This looks like homework.  I've tagged it as such so you can get appropriate answers.  If it's not then you can remove the tag.
The following is comments in mixed model group: 
You might look at the regress and spatialCovariance packages of David Clifford.  The former allows fitting of (Gaussian) mixed models where you can specify the structure of the covariance matrix very flexibly (for example, I have used it for pedigree data).  The spatialCovariance package uses regress to provide more elaborate models than AR1xAR1, but may be applicable.  You may have to correspond with the author about applying it to your exact problem.
@EOL don't be confused, kernel evaluations between two instances are performed no more than once; regardless of the complexity. The absolute number of evaluations depends entirely on how 'easy' a given training problem is.
+1 Worth bearing in mind though that aggressively optimizing the training criterion is likely to lead to over-fitting, unless steps are taken to prevent it, so I would avoid PSO and EA unless the training criterion includes some form of regularisation or other complexity based penalty.
Have you had a look at some of the standard literature on this topic? For example: http://www.stat.columbia.edu/~gelman/arm/missing.pdf and other stuff cited in similar questions here. That might help you narrow your question down to something like 'should I use method x or y to deal with my missing data?'
+1, didn't Fisher have a simple method for combining / averaging p-values? Could that be used here?
You might consider looking into cluster analysis.
values.tmp <- sapply(possibleValues,as.numeric);values.nonnumeric <- values.tmp[is.na(values.tmp)]
Thank you very much for your answers! I will describe my model a little bit better: I am actually trying to see if there is a causality (Granger) between two variables and the yield on certain european 6month T-Bill yields. The first is the spread between 6month EURIBOR and EONIAswap rates. The second is the spread between all-rated 10-year european government debt and AAA-rated 10-year european government debt.
Thank's for the answer and indeed the error was in the code. Everything looks fine now.
How general is the alternative? Even for the general case there are a variety of approaches where you can make a comparison of the fit assuming identical nonparametric functions as against separate ones. Additive models and generalized additive models can deal with such comparisons.
@Ben So N should be 12?
@zeferino see [EDIT 2] thank you
@whuber Also I didn't get the wiki part when it calculates the kriging variance which is similar to the one in your answer. They have the same results but the initial terms are different. How come?
Neither. The x's are assumed known (fixed) as for regression. The mean of the Y's is written as a function ($g$) of the linear predictor, $X\\beta$, but the $X$'s aren't parameters in that, they're constants.
cold-start is precisely the problem you have
One [recent authoritative recommendation](http://pss.sagepub.com/content/22/11/1359) for psychology is no less than 20 items per cell. Animal electrophysiology commonly uses n=2 as a standard. In other contexts, less than 1000 will get you laughed at. More generally, [power analysis](http://en.wikipedia.org/wiki/Statistical_power) can tell you how many items you need.
+1, great answer and graphs!
@ziyuang I'd start with simple global features, like brightness histogram and dominant colors. It's hard to find a representative set of negative samples in such problems, so one should be especially careful not to overfit.
Have you worked out a simple example, such as a case where $X$, $Y$, and $Z$ can attain at most two distinct values each?
It might be a 10K privilege. I can see the deleted answer. It went like this: "You don't need the exact distribution. For the sum of the squares to be at most $a$, each random variable has to be between $-\\sqrt{a}$ and $\\sqrt{a}$. Each satisfies this with some probability $p(a) \\lt 1$. The chance that all $n$ are in that interval is $p(a)^n$ by independence, which goes to $0$ when $n \\to \\infty$, and this is an upper bound for the chance that the sum is at most $a$."  Of course he needs that assumption of unbounded support or support exceeding out beyond a for the argument to work.
You need to compute the Fisher matrix ? what is your model ? I am sure you can make your question more precise.
No.............
I agree. Do you have any data on the claim that there is an underlying structure ? What if someone selects a real sounding name, which is just not his own ?
Thank you for your reading recommendations. The outliers which I want to capture are really erroneous readings, and the sample sizes are between a couple and a few dozens (1000s of variables are in the same data set, each variable has variable number of readings!). I am a bit hesitant to model a distribution for outlier detection, but I will certainly investigate if the measurements distribute the same across the variables.
I like the phrase "Absence of evidence is not evidence of absence" in this respect. Failing to reject would be a case of absence of evidence. Many null hypotheses are of the form "there is no effect", so accepting such a hypothesis would be a case of finding evidence of absence (of a effect).
A big +1. Very thoughtful, clear analysis.
@ArtemKaznatcheev Why can't you just look at a plot of the coefficient series? I was trying to offer a solution that didn't adhere to your desire to test, but as a trade-off was without many assumptions.
You can't say that b/c that is not the overall % of people who felt good, only the OPs friends.
@Shafi Please [register your account](http://stats.stackexchange.com/users/login) -- this way you won't lose control of your question again.
It's the paper that got it wrong.
I'd appreciate if anyone with more skills in latex could edit the calculation in 4. so that it is more readable.
Is this (1) one survey sent to 33 individuals? (2) Are there four questions on the survey? (3) Are the questions binary? likert? (4) the last two lines would be better modified to a statistical question relating to the survey data.
Decision tree is your equation. You can print it out and see it. It will look like a set of conditions. (as @Wake2Sleep duly mentions this is not the best way for continuous variable prediction though). BUT do you really need to see it? If all you need is function connecting two variables all you care is having f(x) and an ability to enter any x in there and find y. All of it can be inside computer, you don't need to understand the equation to use it.
I'd think something like the "For-Against ratio -After" divided by the "For-Against ratio -Before" (essentially an odds ratio) would be a better choice. If it's higher than 1 you improved the odds, if it's less than 1, you didn't.
A cutoff may potentially be useful if there is a discontinuous relationship between $X$ and $Y$ and there is only one potential $X$.  What makes you think the relationship is discontinuous?
(+1) In anticipation of your forthcoming edit. :-)
Haven't you found an acceptable answer on this thread? (check the green mark beneath users' votes)
Well, for one thing, the average income probably does not include children, while the average family size does. Also, is that the average income of people who work, not including unemployed? In that case, if it were "the average number of working people per household", then that reasoning would be sound.
ahh, I was just about to post that! You can also refer to this link for the codes and whatnots: http://www.statmethods.net/advstats/cluster.html
The variates added have a common mean (`0`) and a fixed variance (all identical, also in time). Thinking about it, maybe a single value question would have been easier. But since I will have to deal with linear interpolation in between, I can't get the dimensional dependence out of my head. Also, there will be partial local relaxation. Now I know where to keep looking and what to expect.
Is this question the same as http://stats.stackexchange.com/q/4942/919 ?
Hint: By definition, a [Haar measure](http://en.wikipedia.org/wiki/Haar_measure) is invariant under the action of a group on itself. What group would you (naturally) associate with the circle? What would it mean, geometrically, for a measure to be unchanged under that group action?
I think you still need a multi-level model. It will affect estimation of the variances.
@AlecosPapadopoulos I didn't know forecasts are expected values. Since this forecast is based on regression, how can I see that a regression is simply averaging values? In any case, assuming that we are dealing with averages, then yes, the formula is exactly the same than before. By the way, in your answer you said that this formula ignores various biases, can you describe a bit what you mean?
As a point of departure, it might help you, as well as your respondents, to consult an authority (*not* the Internet!) concerning definitions.  "The *parametric cases* ... are all those in which the class of all [states of nature] can be represented in terms of a vector $\\theta$ consisting of a finite number of real components in a natural way. (...the distribution and loss function depend on $\\theta$ in a reasonably smooth fashion.)  All other problems are called *nonparametric*. --J. C. Kiefer, *Introduction to Statistical Inference,* p. 23.
Thank you for your answer. The next example in the book answers also the question, comparing a model where the clustering is ignored with a correct model and states: "the model based standard errors (assuming no clustering) are misleadingly small for the randomised intervention effects and lead to substantively different conclusions about the effect."
You question is rather tricky... I'm thinking in it.
@Ampleforth, in that paper, I fit a distribution to empirical data, so by "fitted CDF" I meant the theoretical CDF of the fitted distribution. Sorry, I see how I could have been more clear!
Done, at least in outline. Considering the second question is more easily tackled with a pivot of $\\hat\\mu/\\mu$, I may have been better using that from the start and dealing with a slightly more fiddly calculation at the start, to save more fiddling later. I may go back and rework the post with that in mind. To double check: I am correct in thinking that your $\\pm$20% is asking that $\\hat\\mu/\\mu$ lies between 0.8 and 1.2, right?
My study is cross-cultural research, based on Semantic Differential technique. I want to extract factors and compare results for two groups of students (first group from China, second group from Russia).
Maybe this is because I am not a statistician, but when it seems you assumed linear regression when the OP asked about regression in general.  But I guess the gist is that adding covariates to any kind of regressor makes some assumptions about the input space so it can extrapolate to new examples, and matching is more cautious about what kind of things can be extrapolated.
This seems related to [your other recent question](http://stats.stackexchange.com/q/31771/2970). Maybe you should consider updating your other one to clarify your interests, instead. :)
@dato datuashvili - ermm why? Because then you have to take means again?
Limiting yourself to parametric tests needs an explanation.  And normalizing ordinal data will not help when the variable is discrete.
@Baumann Yes they are. The $N$ objects are supposed to be similar.
Worth a quick mention: the link to the University of Cambridge webpage is very good. It contains two very nice examples complete with solutions and graphs.
You're right about the explanation; your reference nailed it. But your situation looks unusual: it appears you have only ten or so *independent* responses (which lie on a continuous scale, not a discrete one) but you are using multiple explanatory variables that vary over time. This is not a situation contemplated by most regression techniques. More information about what these variables mean and how they are measured might help us identify a good analytical approach.
Thanks for clarifying. I did a very large grid search for these parameters so I do not think that is the problem. I am pretty confident that the learning parameters are calculated correctly.
This kind of plot takes a few seconds to make in R, after you've imported your data. Is it important that it's in matlab?
Oh I see. I'm just unsure the major differences between pop and sample.
I would then have to show a pie chart at productivity=1, productivity=2, ..., productivity=5 for each question, right?  I'm looking for less.
It would be nice if you define in your text what are $d$ and $k$. Also, why did you used tag `bayesian`? Describe you sequence - i.e. you first were doing PCA of features and then classifying objects based on the extracted PCs, or other way round?
Possible duplicate: http://stats.stackexchange.com/questions/3395/smoothness-of-a-surface.
You can upload your figure to imgur.com and give the i.imgur.com url and someone will fix it for you. If it's already somewhere online give that URL... but it looks to me like you already have more than 10 rep.
What precisely do you mean by "predictive power"?  For instance, var3 might have extremely high correlation with the dependent variable in a sample but you might need to make predictions where var3 deviates substantially from the values it attained in the sample, thereby introducing huge uncertainty of prediction.  OTOH, var1 and var2 might individually have poor correlations with the DV, but together might work well for the intended predictions.  This indicates that measuring "predictive power" requires a specific *prediction context* in addition to the model.
Can you elaborate a bit? It seems some iterative algorithm (matrix invert, QR decomposition) seem to work well. I'm also curious as to whether the inaccuracy of SP becomes more of a problem for operations involving larger arrays.
@probabilityislogic really neat! And what if $\\nu$ is also unknown? Can you also give some reference please? Maybe best here: http://stats.stackexchange.com/questions/87405/t-distribution-parameter-estimation
This looks like a repeated measures situation. If your only interest is in comparing mean blood sugar levels, I don't think what you did is actually wrong, but it may not be as informative as a model for your collection of observed values.
My problem stemmed from the fact that adding an interaction term into my regression model reduces the tolerance of my predictor variables down to close to zero, so I need to centre the values.  However, I don't need to do this in ANOVA, and in ANOVA if my interaction term isn't significant I just interpret the main effects, but in a multiple regression I include the interaction term in the model even if it's not significant.  I think I'm clearer in my head as to how the analysis actually works in the background now though, so I can see why my p values are affected by adding in the interaction
How were the pilot regions selected? What is the difference between region and resp. nontreated/treated location?
This is NOT enough information to help you. Usually you can't work around zero inflation by just transforming the data. You actually need to model it or use some other sort of appropiate analysis. Telling more about the data (where is it from, how is it distributed) and about your questions might lead to a more helpful answer.
Huh? The predicted y is given by the original formula in your question. The b are estimated by running the regression (e.g. OLS regression) based on your data. You don't calculate b in terms of y; and y shouldn't have subscripts.
You should delete this 'answer' and add it to your question as you have not really found the answer as evident by the last line.
(+1) This provides a nice illustration of what Cover and Thomas refer to as the "strange self-referential character" of the entropy. However, I'm not sure the answer actually addresses (directly) the OP's apparent concerns. :)
More generally the problem in statistics is not that you are unable to prove the null hypothesis, it is that you are not able to make any point estimates with certainty.  That is, just as you can't say "there is no effect of the variable" you are unable to say that "the effect size of the variable is 1.95".  Statistics always have confidence intervals.
What are $X$ and $Y$ and what is their relationship to $x$ and $y$?  What definition of "continuous" are you using?  (There *are* different definitions and they *do* matter for the answer.)
A simple help through the rseek.org or r-help mailing list will get you on your way with dealing with most (all?) of the methods in R (which is the program package I would suggest to anyone). Good link.
What's the question??
There is no information in this question that even suggests a probability model for the two functions, so the short answer is no, you cannot determine a probability.
More relevantly, I do not see how you PROVE that the pairing is sensible by using the test. Showing that with that pairing you get a significant result is certainly not a proof. If I had to reason for the sense of a pairing, I would show that the inner group variability is significantly reduced compared to the overall variability.
One idea is to use beta distribution to model $c=\\frac{a}{y}$ and then set $\\hat{b}=y(1-\\hat{c})$
Thanks. That's seems helpful. I'll look more into it.
Would the [BDS statistic](http://www.princeton.edu/~joses/pp/bds.pdf) be applicable?  If so, there are [reasonably fast implementations](http://scholar.lib.vt.edu/ejournals/SNDE/snde-mirror/002/articles/v2n2003.pdf).
@Ants ITS=Inverse Transform Sampling
No idea how I should apply such a method to my issue, sorry. Upvote for pointing me to `scikit-learn`, I hadn't heard about that package, thanks.
@Luke404: Weka has 3 types of algorithms (classification, clustering and association mining), and they decided to put regression into classification section. But in general you are right, classification and quantitative prediction are a bit different things.
@DavidMarx, I mentioned that this is the software package that I am using, but my question is about "the possibility of getting the probability of a prediction via logistic regression". I downloaded Tanagra yesterday, and happy to use any other software that can do this. :) Thanks for the R link
Let's just say I have 10 readings record, values (x,y,z) the model is w = ax + by + cz, how do I determine a,b,c using the 10 readings? sorry my maths and stats is at beginner stage.
@styfle - I was writing my comment as you wrote yours.  What you may not be aware of is the factorials tend to "pile up" very quickly.  In fact they often grow so fast $n!\\approx O(n^{n})$, that you can often approximate a sum of "choose" functions by the largest one.  You could also convince yourself of this by running some simulations.
Your second equation is not in general true (even approximately).  To see this, consider halving the gaps between the $\\theta_i$ and thereby doubling the number of elements in the sum.  You will, given some smoothness assumptions, get a sum that is approximately twice that of the original sum (2x the number of elements with about the same average value per element.)  Basically, you've left out the "$\\text{d}\\theta$" part of the integral in the approximating sum.
I think you have identified an important source of confusion: in the first line, "$x$ is the number of successes" whereas in the middle it suddenly has changed to a value that is either $0$ or $1$. For instance, when $n=2$ the first meaning of $x$ can have the values $0,1,2$ with probabilities $q^2,2pq,p^2$. The expectation of $x^2$ therefore equals $0^2q^2+1^2(2pq)+2^2p^2=2p(1+p)$: that's not equal to $np=2p$.
Good job!! This is less of a color explosion :D It also draws the emphasis on the diagonal square which show the groups of genes up-regulated or down-regulated in the groups of cells. The information is the same, but I personally find it more intuitive.
I think any distribution that gives a sig. result with N = 6 will be so non normal that it will pass the IOTT with flying colors - that's the interocular trauma test. It hits you between the eyes.
(+1) Also of interest, `smoothScatter {RColorBrewer}` and `densCols {grDevices}`. I can confirm it works pretty well with thousand to million of points from genetic data.
I'd look at a two-component mixture of some continuous family of distributions on the unit interval. (Edit: Uh, like the one Matthew suggested while I was typing.)
But what is your question? The shared variance should be grasped by the canonical correlations.
You may want to read this: [back-transformation-of-regression-coefficients](http://stats.stackexchange.com/questions/27067/).
I'll second Dirk's answer -- that is what I do.  But I do it using multiple populations.  At least in the application that I'm most concerned with, convergence of a single genetic population is not much of an indication of being near the true optimum.
Cohen never intended his proposals for guidelines for quantifying effect sizes to be used as definite categories.
@MatteoFasiolo Thanks! I think my update helps.
It might also help if you give some context, and describe exactly what problem you're trying to solve
You're very welcome @vizzero! In this case, it looks like all of the comparisons of interests are in the model, so I'm not sure what the purpose of post-hoc tests would be. Also, since we see significant interaction, the importance of comparing of the group means (e.g. Group A vs. Group B, ignoring Condition) is not clear to me.
That's funny: when I first saw Python some 10+ years ago I had exactly the same reaction (why is this needed?  Why not just improve what's out there already?  Why learn a whole new set of bizarre syntactic quirks, names of classes, methods, and procedures, and all the rest?).  :-)
Are you expecting all 3 kinds of analyses you requested to have the same values?  Could you say why you imagine this is the case?  A classic ANOVA, lme's ANOVA, and likelihood ratio tests don't do the same things so they don't reach the same values.
Can you show what the heteroskedasticity looks like? What is the original response variable measuring? Have you considered using models that incorporate nonconstant variance, such as simple GLMs? There's also [heteroskedasticity-consistent standard errors](http://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), depending on what you're trying to achieve with your model.
This *may* be a somewhat garbled version of the *data-processing inequality*. Do you have a link to slides from the presentation, or some other detail, by chance? It is a standard fact that conditioning on an extra variable *reduces* entropy, for example.
I think Azeem needs a stronger example. You could object to 2.5 as the average of 1, 2, 3, 4 children per family on the same grounds, how is that to be interpreted as it is not one of the defined values. That raises different issues.
It would help if you posted a reproducible example.  Thanks!
After taking the measurement the subject will no longer be in the study. Is that correct?
What kind of operations do you actually do on the files? In my mind, that dramatically affects the range of acceptable options. Also, it seems to me that you could edit TSV data in a word processor and line up tabs to preserve columns -- as long as your word processor can be set to not do "smart" conversions and can save as plaintext, which many can.
Where is this explained in the video?  It's an hour long.  Thanks!
do you mean that in the multinormal case, both estimators are equivalent?
Hi David - the site you linked to is really great - thank you.