I'm currently running a perl program where I have to take a 1 million line text file, break it down into chunks (anywhere between 50 and 50,000 lines per chunk), and run some calculations and such on them.  Right now, I load all of the data into array1.  I take array2 and use it to pull just the chunks of data I need.  I then do what I need to perform on array 2, and then go back and grab the next set.

example data

A, blah1, blah2

A, blah6, blah7

A, blah4, blah5

B, blah2, blah2

So I would grab the first three into array 2, sort them, then move on to the next set. My program works pretty well and efficiently to begin with, but it experiences a severe slowdown later on.

50K takes 50 seconds, 100k takes 184 seconds, 150k takes 360 seconds, 200k takes 581 seconds, and it only gets exponentially worse as the program continues (4500 seconds at line 500k)

No, I cannot use a database for this project, any suggestions?

my @Rows1=&lt;FILE&gt;;
my $temp = @Rows1;
for($k = 0; $k &lt; $temp; $k++)
{
    my @temp2array = ();
    my $temp2count = 0;
    my $thisrow = $Rows1[$k];
    my @thisarray = split(',', $thisrow);
    my $currcode = $thisarray[0];
    my $flag123 = 0;
    $temp2array[$temp2count] = $thisrow;
    $temp2count++;
    while ($flag123 == 0)
    {
        $nextrow = $tuRows1[$k + 1];
        @nextarray = split(',', $nextrow);
        if ($currcode eq $nextarray[0])
        {
            $temp2array[$temp2count] = $nextrow;
            $k++;
            $temp2count++;
        }
        else
        {
            $flag123 = 1;
        }
    }
}


I have edited my code to more resemble the answer below, and I've got these times:

50k = 42, 100k = 133, 150k = 280, 200k = 467, 250k = 699, 300k = 978, 350k = 1313

Its not exactly keeping linear, and by this trend, this prog will still take 14000+ seconds.  I'll investigate other parts of the code


Question: why do hosting co charge so much just for RAM?
Answer: Because;


Very often you're buying a virtual server,
These companies want as many virtual servers per physical server a possible,
While VMWare/Xen/whatever can easily time-slice all those low-CPU virtual servers even VMWare's memory overcommitment isn't as good as its vCPU equivalent,
So virtual server providers need to stuff their servers with memory,
Servers that take a lot of memory aren't cheap,
Bigger, more reliable, memory chips cost more than small, consumer memory - MUCH more.


and the most important reason - people will pay their rates for it, so why should they stop?