100,000 lines of Typescript: A data-backed retrospective
Tags: typescript, computerscience, programming, webdev
Published: 2021-01-19 17:16:18

## PROLOGUE:

*If you're more interested in the results of the migration than the journey itself, scroll on down to DATA & RESULTS*.

At first, types seemed tedious, and not worth the effort. I remember thinking, back in college, why was I specifying that a string (as indicated by the name `foo_str`) was a string (as indicated by the quotation marks) was a string (as indicated by the type)? Then Typescript came with a revelation. Type inference[^1]! The computer would do the grunt work for you, leaving you free to wander outside and sing and  dance and skip and shout joyfully into the sunlight[^2]. I would never use Javascript again.

## REGEXES EVERYWHERE

I was using Javascript again.

Or to be more accurate, my coworkers at my new company [15Five](https://15five.com) were using Javascript. Those poor souls. I nobly volunteered (was assigned a ticket) to save them from their struggle (they were doing okay, really), by migrating the codebase to Typescript (by migrating the codebase to Typescript).

This was my first time doing a migration of this size, and it was a daunting task. There were over 300 files and literally thousands upon thousands of Typescript errors. I knew migrating it by hand, alone, was out of the question, so I checked to see if there were any auto-migration tools. There was one! Yes! It failed to run! Yes! Wait, No!

Unfortunately, that was the only one I found. Other large migrations I read of were done by [teams of people with a solid chunk of time ](https://www.lucidchart.com/techblog/2017/11/16/converting-600k-lines-to-typescript-in-72-hours/) dedicated solely to the migration, and the assistance of specialized tools.

Fortunately, we already used React [PropTypes](https://reactjs.org/docs/typechecking-with-proptypes.html), which are somewhat similar to Typescript types. In some cases, very similar. A simple search/replace across the entire project (VSCode made this very easy) sufficed to take care of basic cases like converting `PropTypes.string` to `string`. This alone was a massive time saver as I could change ~300 files at once. 

But plain text replacement can still be slow, as you have to go through each scenario one by one. With regex you can handle multiple scenarios at once:

`PropTypes.(string|number|object|symbol|any).isRequired`
Substitute with `$1` to replace all the primitive PropTypes at once!

`PropTypes.arrayOf\((\w+)\).isRequired`
Substitute with `$1[]` to replace one-line array types.

`([^:]+): PropTypes.array,`
Substitute with `$1?: any[];` to replace bare arrays.

Regexes aren't as powerful as [codemods](https://github.com/facebook/jscodeshift), but they were enough for me. Within a week I had a PR merged that converted the entire react codebase to Typescript.

## TSC-SILENT

But I wasn't done. Not by a long shot. Although the PropTypes were in Typescript, and the files were technically Typescript files with a .ts extension, there were still thousands of miscellaneous type errors that polluted a Typescript run. Devs could still have the benefits of Typescript on new code, but on old code it wasn't as powerful. Furthermore, the type errors prevented us from running Typescript in our continuous integration (CI) environment.

Fortunately, I found a tool called tsc-silent, that allowed you to run Typescript while silencing certain types of errors. That way the CI could catch any new errors that appeared, while also giving us a progress report on how many old errors were left. Tsc-silent was highly configurable, letting us silence certain errors in directors of the old codebase while catching all errors in other directories.

Another option I could have done but didn't think of at the time was a snapshot test. By saving Typescript errors to a file and failing CI if the run doesn't match the file, you can catch any new errors introduced and keep track of existing errors. For more details see https://dylanvann.com/incrementally-migrating-to-typescript.

## THE LONG GRIND

For a while I tried tackling the remaining Typescript errors by myself. This went slowly, given my limited time and lack of expertise in the codebase. Eventually, I realized I needed subject matter experts, so I created an every-other-week meeting with a few generous volunteers from the frontend team. Thanks to the expertise of the volunteers, we made good headway at first, but as the months went by each type error took longer and longer to solve. We were so close - only 500 errors left! Yet fixing just a couple errors could take an entire hour. 

For example, certain errors may involve your code's use of a foreign package, so in order to fix it you need a good knowledge of both your code and the package's API. If the library doesn't have good types then the issue can take a while to untangle. The fix may pretty easy (one PR literally just added `<Date>`) but *knowing* how to fix it is a whole other story.

As 2020 neared its end (good riddance) I decided to switch strategies.

## TS-MIGRATE

Airbnb open-sourced their automatic JS->TS migration tool, a project aptly called [ts-migrate](https://github.com/airbnb/ts-migrate). Although our codebase was already migrated, the feature to automatically add ts-expect-error comments to all existing errors seemed useful. Using that to ignore existing errors would let us Typescript directly in our CI instead of tsc-silent and prevent new errors from creeping in. Running it took a while, but soon tsc produced a clean run! I added it to our CI, and a simple grep allowed us to keep track of the remaining type errors.

```yml
- run:
    name: Leftover type errors per file
    command: grep -cR "// @ts-expect-error ts-migrate" your/path/here | grep -v :0
- run:
    name: Number of leftover type errors
    command: grep -R "// @ts-expect-error ts-migrate" your/path/here | wc -l
```

There's now ~200 errors left, but a lot of them will be gone once we refactor out our usage of Redux. I would like to say we got down to 0 errors, as that would make for a neat and tidy ending to this story, but the real world is not so perfect. When Redux is gone we can go through the remaining errors and decide what is worth fixing.

## DATA & RESULTS:

**Team size**:
* 11 Frontend engineers in total (not directly involved with TS)
* 3-4 engineers directly involved in migration
* 1 Project lead (not included in above)

For context, most of our employees are senior or mid-level coders. We have no interns.

**Codebase Size**:
* Jun 19th, 2019: 291 files, 25,866 lines
* Dec 7th, 2020: 1560 files, 126,816 lines

**PR's**: 47

**Person-hours**:
*very rough estimates follow*
* Assuming each PR ~= 1 hour, that's 47 hours
* Plus 10 hours for meetings
* Plus 5 hours for PR reviews
* Total: 62 hours

If we had ts-migrate at the start, that might have saved 9 hours or so.

**Number of TS Errors Fixed**: 3,168

**Time**:
* Jun 19th, 2019: Started work
* Jun 25th, 2019 (6 days): Migrated to TS
* Sep 30th, 2019 (3 months): TS enforced in CI
* Dec 7th, 2020 (1.5 years): Vast majority of errors fixed, TS used directly in CI

Note that most of the benefit of the migration was realized within the first few months. This follows the Pareto Principle - 80% of the outcome results from 20% of the work.

**Bugs**: 0 bugs, thanks to great reviews by [Razh](https://github.com/razh)!

**Survey data**:
* [Mid-way Survey](https://docs.google.com/forms/d/1H0XjPCu5mF2TVllrjHo7HExyUT4Ajl8pqjWBNsfuX3s/viewanalytics)
* [Final Survey](https://docs.google.com/forms/d/e/1FAIpQLSdhYXYOXfQOfqg7-YXPWang3BMrnSlU96GYccb5ba7eXmZ6Zg/viewanalytics)

Although the sample size was too small to be statistically valid, the survey results were very good. Most developers were happy with Typescript. It caught bugs, improved intellisense, and made refactoring easier. The survey did reveal mixed results on making development faster, but that is no surprise. The work it takes to add types somewhat negates the improved development speed from refactoring and intellisense. The benefit you are left with is catching bugs.

## SUMMARY:

Converting to Typescript with ts-migrate is a quick and easy win to improve your frontend code. Fixing all Typescript errors on a large codebase, on the other hand, is not easy nor quick. Going through them all can be a slog, so you should maintain inertia by holding type-fixing meetings or hackathons. Certain type errors may be rather difficult to resolve, so you should decide ahead of time the maximum amount of effort you are willing to spend. You can realize the majority of benefit from Typescript without fixing all the errors, so you don't need to worry about fixing everything in one long all-nighter. You can take your time - so why not start today?

[^1]: I don't mean to suggest Typescript was the first to come about with type inference, just that it was the first to bring it to Javascript at a large scale.
[^2]: Or to stay inside and write articles about how free you are







<!--
Unfortunately, that was the only one I found. Other large migrations I read about in blog posts were done by [teams of people with a solid chunk of time ](https://www.lucidchart.com/techblog/2017/11/16/converting-600k-lines-to-typescript-in-72-hours/) dedicated solely to the migration, with the assistance of automated tools.

When I joined [15Five](https://15five.com) the frontend codebase was in Javascript, so I volunteered to migrate it over to Typescript. 


* length of time (~2 years). Started work on June 19th. Migrated to TS Jun 25th. (see https://github.com/15five/fifteen5/pull/23514). Enforced in CI sep 30th, 2019. 

Should probably gloss over below regex (briefly mention it, provide one or texamples) and just say to use https://dev.to/kojikanao/introduce-typescript-to-react-js-project-with-ts-migrate-190h We used [ts-ignore](https://dev.to/15five/how-to-temporarily-ignore-errors-during-a-typescript-migration-doe) to great success. Gloss over this too cause ts-migrate better.

Data:
the effort necessary to fix them is not much more than the value we get out of it, so we can take our time

Python's simple and elegant syntax made it easy and welcoming for the beginner I was at the time, and I took to it like a duck in water, creating a reddit bot, which in turn landed me my first CS job. But the employer used C#, so I was back in the world of types.

Title ideas: 
* X lines and Y lines to go?
* Migrating 15Five to typescript: a data-backed retrospective
* Migrating ? lines to typescript: a data-backed retrospective
* 100,000 lines of Typescript: a data-backed retrospective

TS usage is steadily growing: https://2019.stateofjs.com/javascript-flavors/typescript/
Lots of people interested!
TS is #2 most loved language: https://stackoverflow.blog/2020/05/27/2020-stack-overflow-developer-survey-results/

I had such a hard time figuring out how to share form results - eventually came across 


customizable, letting us configure it to only ignore certain errors in the old codebase and catch all errors in the fixed sections of the codebase.

https://github.com/15five/fifteen5/pull/24175

Look for `style={\w+}` and type the var with React.CSSProperties
`error TS2322: Type '<T>' is not assignable to type 'CSSProperties'.`

 (checkout https://github.com/15five/fifteen5/commit/56d11140a5357ff830da4e8dca755a2baae3aafd and run tsc to find out number of errors fixed)


https://github.com/15five/fifteen5/pull/24754

https://medium.com/@martin_hotell/react-refs-with-typescript-a32d56c4d315

2-3 engineers in the bi-weekly 1 hour Typescript meetings

for `this.foo=4` in constructor move that out above constructor and make it `private foo=4`. If there's some logic you need to do in the constructor to assign the value there's also a quick fix available for just declaring the property.

^ used above regex to replace instances where defaultProps was assigned to Instead of having defaultProps var directly defaultProps var contents to func.defaultProps (type is inferred automatically as Partial). To find places where defaultProps was assigned to a variable I looked for `.defaultProps = defaultProps`

ex: 
before: `defaultProps={foo:3};fooComponent.defaultProps = defaultProps`
after: `fooComponent.defaultProps = {foo: 3}`

The variable is obviously a string, so why did I have to type it out when the computer could have easily inferred that? This, along with the general bloat of Java, turned me off of types and onto Python. 

For the memo components (for example, `ff/static/modules/usage_dashboard/src/pulse/PulseChartLegend.tsx`) the defaultProps had an type error: "Property 'defaultProps' does not exist on type 'NamedExoticComponent<Props>'.ts(2339)" 

It's related to https://github.com/DefinitelyTyped/DefinitelyTyped/issues/36320

Check out the guide for more (todo: make guide. use regex101? but focus on rest of article first, guide is optional)

https://regex101.com/r/R5gwq3/1

Probably just need to move defaultProps up to inside memo.

https://github.com/15five/fifteen5/pull/25457

https://github.com/DefinitelyTyped/DefinitelyTyped/issues/20544

https://github.com/Microsoft/TypeScript/issues/27552

https://github.com/DefinitelyTyped/DefinitelyTyped/pull/40433

https://15five-dev.atlassian.net/browse/ENG-3048

Great Airbnb video: https://www.youtube.com/watch?v=P-J9Eg7hJwE
https://github.com/danvk/dtsearch  

*If you're more interested in the results of the migration than the journey itself, scroll on down to RESULTS & DATA*

When I started college I loathed types. They seemed clunky, cumbersome and pointless.

`String message = "hello world";`

In one line I declared that the variable was a string, not one, not two, but three times! Once with the quotation marks, again with the type, and a third time implicitly with the variable name. It was overly verbose, just like the rest of Java. So when I found Python, I immediately fell in love:

`message = 'hello world'`

In comparison Python had such a simple and elegant syntax. Even the quotation marks took up less space! For the beginner I was at the time, it was much easier - and more pleasurable - to use. I created a Reddit bot with Python, which in turn landed me my first internship. A job soon followed, but not in Python - C#. I was back in the world of types.

However, types came much smoother this time around. C# was less verbose and easier to use than Java, and I had more experience with programming at this point. Furthermore, Visual Studio offered suggestions for properties based on the type of the variable, an extremely useful feature called intellisense that I could not live without. The olden days of constantly looking up documentation or relying on fallible human memory now seemed so antiquated.

When it came time to design a new product I knew we had to go with Typescript. Angular 2 (yes, it's dated now, don't laugh) with Typescript made frontend easy to work with, similar to C#. I never wanted to work with JS again.

My first encounter with types was not pleasant. 

When I first used types in Java, back in college, they seemed tedious and burdensome. I was happy to stick with Javascript. But Typescript came about, it came with a relavation. 

-->