Bits of Syntax: String Literals
Tags: design, healthydebate, watercooler
Published: 2020-05-04 10:56:21

Have you ever wondered why we can write something like

```
x = 42
```

...in just about any programming language, and the compiler understands what we're trying to say (the number 42), but if we try to save some text to a variable like

```
y = The Answer to the Great Question... is forty-two.
```

...we'll probably get all sorts of syntax errors? The answer has to do with how _string literals_ are defined in code.

## Literals

In most programming languages, decimal numbers are [_literals_](https://en.wikipedia.org/wiki/Literal_(computer_programming)). This means that the source code which defines the number -- in this case the digits `4` and `2` placed next to each other -- is _literally_ interpreted as a number.

"But how could a number be interpreted as anything _other than_ a number?", you might ask. Well, in the vast majority of cases, it can't be. That's why numbers and other characters are treated differently in source code. To explain this difference, let's take a look at alphabetic characters instead of digits.

The character `x` is used above to represent a variable. When we refer to `x` later in the program, we expect it to be replaced, upon execution, with the value that it represents, namely the integer `42`. In this case, `x` is _not_ interpreted _literally_ as the character `x`, but rather _symbolically_, and we -- and the compiler -- understand that symbol `x` to represent that value `42`.

When we create a string literal, we're telling the compiler that we want some chunk of characters to be interpreted literally, and not to try to parse the sequence of characters as source code. So while we can write something like

```
a = 3
b = 4
c = a + b
print(c)
```

...in most languages, and expect that the number `7` will be printed to the console, writing something like

```
a = 3
b = 4
c = "a + b"
print(c)
```

...would instead print the literal sequence of characters `a + b`. By [_delimiting_](https://en.wikipedia.org/wiki/Delimiter) the string with double-quote characters on each side, we're telling the compiler that we don't want it to evaluate the terms on the right-hand side of the third line, but rather we just want to assign to `c` the literal sequence of characters inside the quotes.

Trying to treat literal sequences of characters and literal sequences of digits on an equal footing leads to ambiguities. When you say

```
forty-two
```

...do you mean the literal string of characters "`forty-two`", or do you mean the value represented by the variable named `two`, subtracted from the value represented by the variable named `forty`? And if you said

```
w = "42-19"
```

...are you trying to subtract the number `19` from the number `42`, and assign the result to `w`? Or are you just trying to define a sequence of characters that should be assigned to `w` as-is?

In order to avoid interpreting text as special characters and variable names, we _must_ somehow define where a string literal begins and ends, so the compiler can figure out when it should stop trying to interpret source code characters as language constructs and when it should start again. In essence, a quoted string is really shorthand for a function of the form `literal(...)`, which takes an arbitrary sequence of characters as its only argument and returns them as-is, without any interpretation.

> Another way to think about this is what there are different _modes_ we're typing in when we program. In the "default mode", we want the compiler to interpret everything we write as code -- every `-` is a minus sign, every sequence of `contiguous` alphabetic characters is a variable or function name, and so on. We _shift_ to the "literal mode" with a toggle character, like `"`, and often shift back using the same character. This is not unlike when you hold down the <kbd>shift</kbd> key to temporarily move from lowercase to uppercase letters. 

## Common Methods for Delimiting Strings

So how is this accomplished in different programming languages? In most modern languages, [_typewriter quotes_](https://en.wikipedia.org/wiki/Quotation_mark) aka. "straight" or "dumb" quotes are used to delimit the beginning and end of a string

```
"like so"
```

Using the same character [(ASCII #34)](https://en.wikipedia.org/wiki/ASCII) to mark both the start and the end of the string can also lead to difficulties when the delimiting character is embedded in the string itself (known as [delimiter collision](https://en.wikipedia.org/wiki/String_literal#Delimiter_collision))

```
"where does "this" string start and end?"
```

Many languages allow delimiters to be [_escaped_](https://en.wikipedia.org/wiki/Escape_sequence) -- either by doubling (`"like ""this"""`) or by preceding with a backslash `\` or carat `^` character (`"like \"this\""` or `"like ^"this^""`). Some languages (like [JavaScript](https://www.w3schools.com/jsref/jsref_obj_string.asp) and [Python](https://www.w3schools.com/python/python_strings.asp)) also allow apostrophes to delimit strings, which means strings can be nested at most two levels deep without escape sequences

```
"like 'in this' example"
```

But [some languages use different characters](https://en.wikipedia.org/wiki/String_literal#Paired_delimiters) to mark the _beginning_ and _end_ of a string, like

- [PostScript](https://www.tinaja.com/glib/strconv.pdf), which uses open-and-close parentheses to delimit strings `(...)`,
- [m4](https://en.wikipedia.org/wiki/M4_(computer_language)), which uses a backtick-apostrophe pair `` `...' ``<sup>*</sup>, and
- [Tcl](https://gerardnico.com/lang/tcl/curly), which allows open-and-closed braces to delimit strings `{...}`

> \* If you're wondering how I did this in DEV's Markdown editor, you can define inline code blocks using _two_ backticks instead of a single one. Just make sure you put whitespace around the double-backtick delimiters, like so: ``` `` `...' `` ```

Having different characters for beginning and ending a string means that the opening character no longer needs to be escaped within the string, but the closing character must be treated specially. For example, for a string delimited by open-and-close parentheses,

```
does (this (string end here?) or here?)
```

If we truly ignore all non-`(` characters within the string, then it should end at the first `)` character. But if we want to allow nested strings (or nested comments, [notoriously unavailable in HTML](https://stackoverflow.com/questions/18145886/html-nested-comments)), our parser needs to track all non-escaped `(` and `)` characters within the string, and keep track of how many "levels deep" we are, and things quickly get messy.

## Uncommon Methods for Delimiting Strings

Other languages have tackled this problem in interesting and unusual ways. [FORTRAN](https://en.wikipedia.org/wiki/Hollerith_constant) avoided the delimiter problem altogether by using Hollerith notation, in which the length of the string, in characters, preceded the string itself, followed by a literal `H` character:

```FORTRAN
33HI am a string with 33 characters.
```

This method, of course, is clunky and error-prone when characters must be manually counted by programmers, or if using characters beyond ASCII, which require more than 1 byte per character.

Some languages allow [whitespace-delimited strings](http://blogs.perl.org/users/tinita/2018/03/strings-in-yaml---to-quote-or-not-to-quote.html), like YAML:

```yaml
myString: |
  This is a long string, containing various special characters,
  like " < \" ^ ) }
  This is fine.
```

In a YAML _literal block scalar_, like the one above, the indentation defines the scope of the string. In [MediaWiki template parameters](https://en.wikipedia.org/wiki/String_literal#Whitespace_delimiters), one can define a string using newlines as delimiters, like so:

```
{{Navbox
 |name=Nulls
 |title=[[wikt:Null|Nulls]] in [[computing]]
 }}
```

The above string `name` contains the sequence of five characters `Nulls` as its value.

Finally, in some languages, in restricted contexts, strings can be inferred with no delimiters at all, like when [defining property names in a JavaScript object](https://mathiasbynens.be/notes/javascript-properties):

```js
var myObj = {
  red: 0x00f,
  blue: 0x0f0,
  green: 0xf00
}
```

Above, `red`, `blue`, and `green` are _strings_, not variables, but in this narrow context, the quotes can be omitted. (JavaScript objects defined in JSON data should _always_ have property names quoted, however.)

## Conclusion

A _literal string_ is really just a sequence of characters in source code which we _do not_ want to be interpreted. It should be treated literally, as-is (as just a sequence of characters).

To demarcate the beginning and end of this "no interpretation zone", however, we need some kind of flag, or signal, to let the compiler know where we want code interpretation to temporarily pause, and where we want it to pick back up again.

This is made more difficult by the fact that those flags are embedded in the source code itself, using the same medium (characters) as the rest of the source code and the literal characters contained within the string.

This means that we can't be completely insensitive to the characters which appear after the delimiter at the beginning of the string, we need to step through the source, character-by-character, "watching" for the delimiter that ends the string. This leads to issues with nested strings, embedded delimiters, and more.

Maybe someday, someone will come up with an easier, more intuitive and robust, less error-prone method than the one we currently have for embedding literal sequences of characters within source code. But for now, literal strings in code seem to be held together by an ad-hoc collection of duct tape, escape sequences, and, well, string.

---
---

Follow me: [Dev.To](https://dev.to/awwsmm) | [Twitter.com](https://twitter.com/_awwsmm)
Support me: [Ko-Fi.com](https://ko-fi.com/awwsmm)

Thanks for reading!