Repository: go
Issue: proposal: encoding/json: add "readonly" tag
Labels: Proposal,Proposal-Hold

Previous discussion: #19423

Note: this proposal focuses on the encoding/json package, but the same could be
applied to other encoding/\* packages, especially encoding/xml.

Problem
-------

It is currently hard to marshal json while preventing those fields being set to
arbitrary values by outside input. The `-` tag prevents both unmarshalling and
marshalling.

This is a common requirement in for example REST APIs:

	type User struct {
		ID        int       `json:"id"`
		Name      string    `json:"name"`
		Email     string    `json:"email"`
		CreatedBy int       `json:"createdBy"`
		CreatedAt time.Time `json:"createdAt"`

		DefaultUnmarshal bool `json:"-"
	}

When showing the data to the user (e.g. `GET /user/1`) we want to marshal the
`ID`, `CreatedBy`, and `CreatedAt` fields; but we *don't* want to allow users to
set these fields on create or update endpoints (e.g. `PUT /user/1`).

As far as I can think of, right now it's hard to write a generic fully correct
solution for this:

1. It's possible to implement the `json.Unmarshaler` interface like so:

		func (u *User) UnmarshalJSON(data []byte) error {
			type Alias User
			var alias Alias
			err := json.Unmarshal(data, &alias)
			if err != nil {
				return err
			}

			if u.DefaultUnmarshal {
				*u = User(alias)
				return nil
			}

			// Unmodifiable fields
			alias.ID = u.ID
			alias.CreatedAt = u.CreatedAt
			alias.CreatedBy = u.CreatedBy

			*u = User(alias)
			return nil
		}

	While this works, it's verbose and difficult to generalize without
	reflection complexity. Every type needs to have its own `UnmarshalJSON()`
	(and/or `UnmarshalXML()`, etc.)

    A second problem is that in some cases you do want to unmarshal the readonly
    fields, for example from tests:

		func TestUser(t *testing.T) {
			body := callAPI()

			var u user
			u.DefaultUnmarshal = true
			json.Unmarshal(body, &u)

			if !expectedUser(u) {
				t.Error()
			}
		}

    Hence the `DefaultUnmarshal` parameter, which needs to be exported to allow
    setting them in other packages.

2. rsc proposed creating a custom `ReadOnlyString` with a no-op
   `UnmarshalJSON()` in #19423. this works well and is not unreasonable, but if
   you want to use e.g. `sql.NullString`; you would need to add a custom
   `ReadOnlyNullString` as well. Using it in combination with the
   [github.com/guregu/null](https://github.com/guregu/null) package means
   creating *two* extra types (`readOnlyNullString` and `readOnlyZeroString`).

   It also doesn't allow easy setting readonly fields from tests.

3. A third option is to unmarshal the JSON to `interface{}`, modify the
   data based on the struct tags, marshal back to JSON, and then unmarshal in to
   the type you want.

   I have a working implementation of this, and it works for the simple cases.
   But the "Unmarshal -> Marshal -> Unmarshal" dance seems needlessly
   complicated and inefficient, and dealing with arbitrary JSON is rather
   tricky/verbose in Go.

4. Another solution (which is probably the most common) is to simply use
   different structs for different purposes. I am not hugely in favour of this,
   as it leads to a lot of duplication.

Proposed solution: `readonly` struct tag
----------------------------------------

With the `readonly` tags added, the above struct would look like:

	type User struct {
		ID        int       `json:"id,readonly"`
		Name      string    `json:"name"`
		Email     string    `json:"email"`
		CreatedBy int       `json:"createdBy,readonly"`
		CreatedAt time.Time `json:"createdAt,readonly"`
	}

Regular `Unmarshal()` will not set any of the `readonly` fields; they are
silently ignored:

	json.Unmarshal(data, &u)

An option for `json.Decoder` can be added to allow setting the `readonly`
fields, similar to `DisallowUnknownFields()`:

	d := json.NewDecoder(r)
	d.AllowReadonlyFields()
	d.Decode(&u)

The `DisallowUnknownFields()` option can be used to error out when readonly
fields are attempted to be set (although this could potentially also be a new
option).

---

In the previous discussion rsc mentioned that this feature would not "pull its
weight" as it's not common enough of a use case. It's true that it's *less*
common of a use case, but I don't believe it's terrible uncommon, either.

Implementing this correctly by users is fairly complex, and adding this feature
to the encoding/json package seems – unless I am mistaken – quite simple to the
point of being almost trivial. It will of course increase maintenance burden in
the future, but personally, I think it's a fair trade-off.

Prototype implementation
------------------------

To test the feasibility and complexity of this change I wrote an implementation
of this proposal, which seems to work well. I can make a CL with an expanded
version of this if this proposal is received well.

<details>

```go
diff --git i/src/encoding/json/decode.go w/src/encoding/json/decode.go
index fd2bf92dc2..e8e2cd7486 100644
--- i/src/encoding/json/decode.go
+++ w/src/encoding/json/decode.go
@@ -273,6 +273,7 @@ type decodeState struct {
	savedError            error
	useNumber             bool
	disallowUnknownFields bool
+	allowReadonlyFields   bool
 }
 
 // readIndex returns the position of the last byte read.
@@ -695,6 +696,7 @@ func (d *decodeState) object(v reflect.Value) error {
		// Figure out field corresponding to key.
		var subv reflect.Value
		destring := false // whether the value is wrapped in a string to be decoded first
+		readOnly := false // ,readonly tag
 
		if v.Kind() == reflect.Map {
			elemType := t.Elem()
@@ -719,6 +721,9 @@ func (d *decodeState) object(v reflect.Value) error {
			if f != nil {
				subv = v
				destring = f.quoted
+				if !d.allowReadonlyFields {
+					readOnly = f.readOnly
+				}
				for _, i := range f.index {
					if subv.Kind() == reflect.Ptr {
						if subv.IsNil() {
@@ -757,7 +762,9 @@ func (d *decodeState) object(v reflect.Value) error {
		}
		d.scanWhile(scanSkipSpace)
 
-		if destring {
+		if readOnly {
+			_ = d.value(reflect.Value{})
+		} else if destring {
			q, err := d.valueQuoted()
			if err != nil {
				return err
diff --git i/src/encoding/json/decode_test.go w/src/encoding/json/decode_test.go
index b84bbabfcd..74b5c7eccc 100644
--- i/src/encoding/json/decode_test.go
+++ w/src/encoding/json/decode_test.go
@@ -2239,3 +2239,46 @@ func TestUnmarshalPanic(t *testing.T) {
	Unmarshal([]byte("{}"), &unmarshalPanic{})
	t.Fatalf("Unmarshal should have panicked")
 }
+
+func TestReadonly(t *testing.T) {
+	type nested struct {
+		RO string `json:"ro,readonly"`
+		RW string `json:"rw"`
+	}
+
+	type foo struct {
+		RO     string `json:"ro,readonly"`
+		RW     string `json:"rw"`
+		Nested nested `json:"nested"`
+	}
+
+	f := foo{"hello", "hello", nested{"hello", "hello"}}
+	data := `{"ro": "XXXXX", "rw": "XXXXX", "nested": {"ro": "XXXXX", "rw": "XXXXX"}}`
+
+	t.Run("unmarshal", func(t *testing.T) {
+		want := foo{"hello", "XXXXX", nested{"hello", "XXXXX"}}
+		err := Unmarshal([]byte(data), &f)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if !reflect.DeepEqual(f, want) {
+			t.Errorf("\ngot:  %#v\nwant: %#v", f, want)
+		}
+	})
+
+	t.Run("allowReadonlyFields", func(t *testing.T) {
+		want := foo{"XXXXX", "XXXXX", nested{"XXXXX", "XXXXX"}}
+		d := NewDecoder(strings.NewReader(data))
+		d.AllowReadonlyFields()
+		err := d.Decode(&f)
+
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if !reflect.DeepEqual(f, want) {
+			t.Errorf("\ngot:  %#v\nwant: %#v", f, want)
+		}
+	})
+}
diff --git i/src/encoding/json/encode.go w/src/encoding/json/encode.go
index f10124e67d..944be253eb 100644
--- i/src/encoding/json/encode.go
+++ w/src/encoding/json/encode.go
@@ -1040,6 +1040,7 @@ type field struct {
	typ       reflect.Type
	omitEmpty bool
	quoted    bool
+	readOnly  bool
 
	encoder encoderFunc
 }
@@ -1156,6 +1157,7 @@ func typeFields(t reflect.Type) []field {
						index:     index,
						typ:       ft,
						omitEmpty: opts.Contains("omitempty"),
+						readOnly:  opts.Contains("readonly"),
						quoted:    quoted,
					}
					field.nameBytes = []byte(field.name)
diff --git i/src/encoding/json/stream.go w/src/encoding/json/stream.go
index 7d5137fbc7..14463f6842 100644
--- i/src/encoding/json/stream.go
+++ w/src/encoding/json/stream.go
@@ -41,6 +41,8 @@ func (dec *Decoder) UseNumber() { dec.d.useNumber = true }
 // non-ignored, exported fields in the destination.
 func (dec *Decoder) DisallowUnknownFields() { dec.d.disallowUnknownFields = true }
 
+func (dec *Decoder) AllowReadonlyFields() { dec.d.allowReadonlyFields = true }
+
 // Decode reads the next JSON-encoded value from its
 // input and stores it in the value pointed to by v.
 //
```

</details>