Glitch free 1,000,000 record data processing in TypeScript with js-coroutines
Tags: typescript, react, javascript, webdev
Published: 2020-07-10 02:38:16

Sometimes we need to process data on the front end, perhaps we are using an offline system or accessing local data.  When that data gets large it can easily cause the UI to glitch.  A few days ago I wrote an [article](https://dev.to/miketalbot/search-as-you-type-at-60fps-with-js-coroutines-35pc) demonstrating how search could be made to run at the same time as UI updates using js-coroutines.  I thought I'd dive into a more powerful version in TypeScript that does more than search; it also renders the records as it goes and has a variety of progress indicators.  Once done it performs a bunch of tabulations to update some charts.

Notice how you can keep typing and even start browsing the records as the searches continue.  This is done using collaborative multitasking on the main thread.

{% codesandbox 4ntsu %}
*Please note the tooltip supplied by Recharts doesn't work properly when this window is zoomed.  See full screen version*

This demo uses a new feature of [js-coroutines](http://js-coroutines.com) that allows you to define a "singleton" function.  Singleton functions automatically cancel the previous run if it is still underway and start again.  That's exactly what you need for a search like this.

```tsx
const process = singleton(function*(resolve: Function, search: string, sortColumn: string) {
    let yieldCounter = 0

    if (!search.trim() && !sortColumn?.trim()) {
        resolve({ data, searching: false })
        addCharts(data)
        return
    }

    resolve({ searching: true, data: [] })
    let parts = search.toLowerCase().split(" ")
    let i = 0
    let progress = 0

    let output : Data[] = []
    for (let record of data) {
        if (
            parts.every(p =>
                record.description
                    .split(" ")
                    .some(v => v.toLowerCase().startsWith(p))
            )
        ) {
            output.push(record)
            if (output.length === 250) {
                resolve({data: output})
                yield sortAsync(output, (v : Data)=>v[sortColumn])
            }
        }
        let nextProgress = ((i++ / data.length) * 100) | 0
        if (nextProgress !== progress) resolve({ progress: nextProgress })
        progress = nextProgress
        yield* check()
    }
    resolve({sorting: true})
    yield sortAsync(output, (v : Data)=>v[sortColumn])
    resolve({sorting: false})
    resolve({ searching: false, data: output })
    addCharts(output)

    function* check(fn?: Function) {
        yieldCounter++
        if ((yieldCounter & 127) === 0) {
            if (fn) fn()
            yield
        }
    }
}, {})
```

This routine starts off by checking if we are searching for something and takes a quicker path if we aren't.  

Presuming it is searching it uses a neat trick of resolving values many times to update the progress.  This allows it to reveal results as soon as it has 250 records, update progress every 1% and then switch on and off searching and sorting indicators.  

Calling resolve just merges some data into a standard React.useState() which redraws the UI to keep everything smoothly updating while the search progresses.

```tsx
interface Components {
    data?: Array<Data>
    searching?: boolean
    progress?: number,
    sorting?: boolean,
    charts?: []
}

function UI(): JSX.Element {
    const [search, setSearch] = React.useState("")
    const [sortColumn, setSortColumn] = React.useState('')
    const [components, setComponents] = React.useState<Components>({})
    React.useEffect(() => {
        setComponents({ searching: true })
        // Call the singleton to process
        process(merge, search, sortColumn)
    }, [search, sortColumn])
    return (
        <Grid container spacing={2}>
            <Grid item xs={12}>
                <TextField
                    fullWidth
                    helperText="Search for names, colors, animals or countries.  Separate words with spaces."
                    InputProps={{
                        endAdornment: components.searching ? (
                            <CircularProgress color="primary" size={"1em"} />
                        ) : null
                    }}
                    variant="outlined"
                    value={search}
                    onChange={handleSetSearch}
                    label="Search"
                />
            </Grid>
            
                <Grid item xs={12} style={{visibility: components.searching ? 'visible' : 'hidden'}}>
                    <LinearProgress
                        variant={components.sorting ? "indeterminate": "determinate"}
                        value={components.progress || 0}
                        color="secondary"
                    />
                </Grid>
            
            <Grid item xs={12}>
                <RecordView sortColumn={sortColumn} onSetSortColumn={setSortColumn} records={components.data} />
            </Grid>
            {components.charts}
        </Grid>
    )
    function merge(update: Components): void {
        setComponents((prev: Components) => ({ ...prev, ...update }))
    }
    function handleSetSearch(event: React.ChangeEvent<HTMLInputElement>) {
        setSearch(event.currentTarget.value)
    }
}
```

The `merge` function does the work of updating things as the routine progresses, and as we've defined a "singleton" function, it is automatically stopped and restarted whenever the search or sort properties change.

The charts each individually start a calculation, and we "join" their execution to the main process so that restarting the main process will also restart the chart.

```tsx
function Chart({data, column, children, cols} : {cols?: 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12, data: Array<Data>, column: (row: any)=>string, children?: any}) {
    const [chartData, setData] = React.useState()
    React.useEffect(()=>{
        const promise = run(count(data, column))

        // Link the lifetime of the count function to the
        // main process singleton
        process.join(promise).then((result: any)=>setData(result))

    }, [data, column])
    return <Grid item xs={cols || 6}>
        {!chartData ? <CircularProgress/> : <ResponsiveContainer width='100%' height={200}>
            <BarChart data={chartData}>
                <CartesianGrid strokeDasharray="3 3" />
                <XAxis dataKey="name" />
                <YAxis />
                <Tooltip />
                <Bar dataKey="value" fill="#8884d8">
                    {children ? children(chartData) : null}
                </Bar>
            </BarChart>
            </ResponsiveContainer>}
        </Grid>
}
```

Here we've use a mix of helper Async functions and generators so we have maximum control.  Our final remaining generator of interest is the one that calculates the chart results:


```tsx
function * count(data: Data[], column: (row: Data)=>string, forceLabelSort?: boolean) : Generator<any, Array<ChartData>, any> {
    const results = yield reduceAsync(data, (accumulator: any, d: Data)=>{
        const value = column(d)
        accumulator[value] = (accumulator[value] || 0) + 1
        return accumulator
    }, {})
    let output : Array<ChartData> = []
    yield forEachAsync(results, (value: number, key: string)=>{
        key && output.push({name: key, value})
    })
    if(output.length > 20 && !forceLabelSort) {
        yield sortAsync(output, (v:ChartData)=>-v.value)
    } else {
        yield sortAsync(output, (v:ChartData)=>v.name)
    }
    return output
}
```

This one simply counts the labels extracted by a function and then sorts the results appropriately.


[![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qc7xww7co3re0jr8weh1.png)](http://js-coroutines.com)