The order of importance of the many components that would make up your grad school app:

1. Research experience --- The most important factor, though not essential. Not having it is not a handicap. In fact most undergrads admitted for grad school do not have research experience, and their app focuses on no. 2 and 3 below. However if you have done some significant reseach, that fact will dominate the rest of your app, and make 2 and 3 irrelevant to a large degree.

2. Recommendation letters --- there are a couple of factors here. A letter from a better known researcher is more valuable. A letter from a professor who is known to your potential advisor is more valuable. Finally, a letter from a professor who has a history of recommending strong candidates carries more weight (admission committees keep track of such things).

3. Undergrad school and GPA --- A 3.6 from MIT is more valuable than a 4.0 from a second tier school. Your GPA over your junior and senior years carries more weight than the GPA over your first two years. This means you can't afford to let up, but you can make up for the follies of youth (or freshman year).

4. Your statement of purpose (the essay you write explaining why you want to pursue a phd in a particular dept) --- This should be focused

5. GRE scores. These are much less important, and are used mostly as a preliminary filtering mechanism.

Another important issue is that given two applications of equal quality (that is, the two candidates are roughly the same by the criteria above), the more focused one will have an edge. By focus, I mean that between an app saying "I'm interested in theoretical computer science" and one saying "I'm interested in probabilistically checkable proofs and error correcting codes" (these are specific topics of research in theoretical CS), the second one will make the cut every time.

As an applicant from a "second tier" school, you are significantly handicapped. The good news is that this system can be hacked quite well. The best course of action depends on how much time you have left before applying for grad school and the subject you plan to apply in, but I would strongly suggest the following:

a) Increase your gpa as much as possible, between now and when you apply. If your gpa is low right now, showing a substantial improvement just before you apply sends a strong message of maturity.

b) Do research. Find a project in your school to get involved in. Find an informal advisor. If you feel such opportunities don't exist in your school, then find the closest top-tier school in your area. Go visit and find a project there. The goal of this involvement would be to produce a publication and a couple of recommendations. Ideally you should do it in conjunction with point c below.

c) Focus your application. This is the single biggest factor that can boost your chances. Decide the subfield within your major that you want to apply to. Find potential advisors within that area. Read up on their research, contact them about it. Do not spam. If possible go meet these people and get involved. If a professor wants to take you on as a grad student and has funding, then nothing on your application matters and you are in.

Its very possible to get admitted to a top tier program. It takes hard work and knowing how the admission process works. You should also talk to people in your field, as some things vary across disciplines.