I'm currently running a perl program where I have to take a 1 million line text file, break it down into chunks (anywhere between 50 and 50,000 lines per chunk), and run some calculations and such on them.  Right now, I load all of the data into array1.  I take array2 and use it to pull just the chunks of data I need.  I then do what I need to perform on array 2, and then go back and grab the next set.

example data

A, blah1, blah2

A, blah6, blah7

A, blah4, blah5

B, blah2, blah2

So I would grab the first three into array 2, sort them, then move on to the next set. My program works pretty well and efficiently to begin with, but it experiences a severe slowdown later on.

50K takes 50 seconds, 100k takes 184 seconds, 150k takes 360 seconds, 200k takes 581 seconds, and it only gets exponentially worse as the program continues (4500 seconds at line 500k)

No, I cannot use a database for this project, any suggestions?

my @Rows1=&lt;FILE&gt;;
my $temp = @Rows1;
for($k = 0; $k &lt; $temp; $k++)
{
    my @temp2array = ();
    my $temp2count = 0;
    my $thisrow = $Rows1[$k];
    my @thisarray = split(',', $thisrow);
    my $currcode = $thisarray[0];
    my $flag123 = 0;
    $temp2array[$temp2count] = $thisrow;
    $temp2count++;
    while ($flag123 == 0)
    {
        $nextrow = $tuRows1[$k + 1];
        @nextarray = split(',', $nextrow);
        if ($currcode eq $nextarray[0])
        {
            $temp2array[$temp2count] = $nextrow;
            $k++;
            $temp2count++;
        }
        else
        {
            $flag123 = 1;
        }
    }
}


I have edited my code to more resemble the answer below, and I've got these times:

50k = 42, 100k = 133, 150k = 280, 200k = 467, 250k = 699, 300k = 978, 350k = 1313

Its not exactly keeping linear, and by this trend, this prog will still take 14000+ seconds.  I'll investigate other parts of the code


Question: User @include url Precedence over Script @include url
Answer: I was told and verified that with scriptish extension it is possible to solve.
This topic.

I hope the greasemonkey developers implement this.
So the best answer so far would be, migrate to scriptish and use the check box that disables script include patterns. Unfortunately, I've already experimented scriptish in past and didn't familiarize.

While greasemonkey doesn't has a solution, I'm going to use both, scriptish only for the scripts with @include http://*