35. Saving data

The servers so far remember nothing. Each request starts fresh, and a form's value is gone when the handler ends. This chapter keeps data in memory, runs into a real problem with that, and then saves the data to a file so that it survives a restart.

Where the data can live

A variable inside a handler lasts for one request, as Chapter 21 explained. To keep data between requests, it must live outside the handlers:

var notes = []note{}

Every handler can now see the same slice. That raises a question that no earlier program in this book had to ask.

Requests happen at the same time

Your earlier programs did one thing at a time. A server does not. net/http runs each request at the same time as the others. Two visitors who click at the same moment are served at the same moment, by the same handler function, using the same package-level variables.

Local variables are safe, because each call gets its own. Shared variables are not. Take a counter:

visits = visits + 1

That looks like one step, but the computer reads the value, adds 1, and stores it. Here is a trace, as in Chapter 04, of two requests when visits starts at 10:

Two visits, and the count went up by one. This is called a data race. It is hard to find, because it happens only when the timing is unlucky. With one visitor clicking, you will never see it. A test with the protection below removed, sending 300 requests with 16 at a time, gave You are visitor number 297. when the right answer was 301, and the number differs from run to run. A slice is worse than a counter: two append calls can lose a message, or the result can be something that neither one wrote.

The earlier chapters did not need any of this, because their handlers only read values that never change. Sharing is safe when nothing changes. From this chapter on, handlers change data, so that data needs protecting.

The mutex rule

A mutex from the sync package is a lock that only one request can hold at a time. Anyone else who asks for it waits until it is free:

// mu protects visits.
var mu sync.Mutex
var visits = 0

func nextVisit() int {
    mu.Lock()
    defer mu.Unlock()

    visits = visits + 1
    return visits
}

Now the trace above cannot happen, because request B cannot read until request A has finished. Here is the rule to follow in this book:

1. Put the mutex next to the data it protects, with a comment that says so.
2. Touch the data only in small functions that start with these two lines: mu.Lock() and defer mu.Unlock().
3. Handlers call those functions, and never touch the data or the mutex themselves.

The rule covers reading as well as writing, because a read can overlap with another request's write. There is much more to say about running things at the same time, and a whole subject beyond this book. This small rule is enough for every program here.

defer

defer puts off a call until the surrounding function ends, however it ends. This is exercises/35/01-defer.go:

// Chapter 35 example: defer runs a call when the function ends.
// Run with: go run exercises/35/01-defer.go

package main

import "fmt"

func check(n int) {
    fmt.Println("Start")
    defer fmt.Println("End")

    if n < 0 {
        fmt.Println("Negative, so return early")
        return
    }
    fmt.Println("Not negative")
}

func main() {
    check(5)
    check(-5)
}

Output:

Start
Not negative
End
Start
Negative, so return early
End

End prints last both times, even on the early return. That is why it suits a lock. defer mu.Unlock() sits right after mu.Lock(), and the lock is released on every path out of the function. Without it, one forgotten Unlock before one early return would freeze the server.

Notes in memory

Here is a working notes page, exercises/35/02-notes-in-memory.go. The part that deals with the data:

// mu protects notes. Any code that reads or changes notes must hold mu.
var mu sync.Mutex
var notes = []note{}

// allNotes returns a copy, so that the caller does not need mu to read it.
func allNotes() []note {
    mu.Lock()
    defer mu.Unlock()

    return slices.Clone(notes)
}

func addNote(n note) {
    mu.Lock()
    defer mu.Unlock()

    notes = append(notes, n)
}

slices.Clone makes a new slice with the same items. The handler can then take its time over building the page, using its own copy, after the lock has been released.

The handlers look like those in Chapter 33 and Chapter 34: POST /notes reads, checks, calls addNote, and redirects. GET / passes allNotes() to the template. Text from visitors is shown by html/template, so it is escaped. Try it:

curl.exe -i -d "text=Buy rice" http://localhost:8080/notes
HTTP/1.1 303 See Other
Location: /

curl.exe http://localhost:8080/
<li>Buy rice</li>

curl.exe -i -d "text=" http://localhost:8080/notes
HTTP/1.1 400 Bad Request
The note is empty.

Now stop the server with Ctrl + C, start it again, and reload:

curl.exe http://localhost:8080/
<li>No notes yet.</li>

The notes have gone. A variable lives in the memory of a running program, and when the program ends, so does the memory. To keep data, it has to go in a file.

JSON

A file holds text or bytes, not structs. JSON is a common text format for data, and encoding/json converts in both directions. This is exercises/35/03-json-basics.go, which has no server and writes no file:

// Chapter 35 example: turning a slice of structs into JSON and back.
// This program has no server, and writes no file.
// Run with: go run exercises/35/03-json-basics.go

package main

import (
    "encoding/json"
    "fmt"
)

type note struct {
    Text string
}

func main() {
    notes := []note{{Text: "Buy rice"}, {Text: "Call Arif"}}

    data, err := json.MarshalIndent(notes, "", "  ")
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Println(string(data))

    var loaded []note
    err = json.Unmarshal(data, &loaded)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Println(loaded)

    err = json.Unmarshal([]byte("this is not JSON"), &loaded)
    if err != nil {
        fmt.Println("Error:", err)
    }
}

Output:

[
  {
    "Text": "Buy rice"
  },
  {
    "Text": "Call Arif"
  }
]
[{Buy rice} {Call Arif}]
Error: invalid character 'h' in literal true (expecting 'r')

• json.MarshalIndent turns a value into JSON, with line breaks and an indent. It returns a []byte, a slice of raw bytes, and an error. string(data) makes it printable.
• json.Unmarshal goes the other way. It takes bytes, and a pointer to the variable to fill, because it must change your variable, as in Chapter 27.
• It gives an error for text that is not JSON. The strange wording comes from the t of this: it hoped for the word true.

Fields must be exported here too. With a field called text, the package cannot see it, and it gives no error. It writes {} for each note, and your data is lost with no warning. So use capital letters on any struct you save.

Reading and writing a file

Two functions from os do the work. os.ReadFile returns the whole file as bytes, and an error. os.WriteFile writes bytes to a file, replacing what was there. Here is exercises/35/04-notes-file.go, again with no server. Each run adds one note:

// Chapter 35 example: loading and saving a JSON file, with no server. Each run
// adds one note. It reads and writes notes.json in the folder you run it from.
// Run with: go run exercises/35/04-notes-file.go

package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "os"
)

const notesFile = "notes.json"

type note struct {
    Text string
}

// loadNotes reads the file. A file that does not exist yet means no notes, which
// is normal on the first run. Any other problem is an error.
func loadNotes() ([]note, error) {
    data, err := os.ReadFile(notesFile)
    if errors.Is(err, os.ErrNotExist) {
        return []note{}, nil
    }
    if err != nil {
        return nil, err
    }

    var loaded []note
    err = json.Unmarshal(data, &loaded)
    if err != nil {
        return nil, fmt.Errorf("%s is not valid: %v", notesFile, err)
    }
    return loaded, nil
}

// saveNotes writes a temporary file, then renames it over the real one.
func saveNotes(list []note) error {
    data, err := json.MarshalIndent(list, "", "  ")
    if err != nil {
        return err
    }

    temporary := notesFile + ".tmp"
    err = os.WriteFile(temporary, data, 0644)
    if err != nil {
        return err
    }
    return os.Rename(temporary, notesFile)
}

func main() {
    notes, err := loadNotes()
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Println("Notes found:", len(notes))

    notes = append(notes, note{Text: "Another note"})

    err = saveNotes(notes)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Println("Notes saved:", len(notes))
}

It reads and writes notes.json in the folder you run it from, which is go-work if you follow the usual command. Two runs:

go run exercises/35/04-notes-file.go
Notes found: 0
Notes saved: 1

go run exercises/35/04-notes-file.go
Notes found: 1
Notes saved: 2

Three parts of this deserve attention.

A missing file is normal. On the first run there is no file yet, and that is not a failure. errors.Is(err, os.ErrNotExist) asks whether this error means "the file does not exist". It treats that one case as an empty list. Any other error, such as no permission to read the file, is still reported.

A broken file is not normal. If the file is damaged, the program must not carry on with an empty list, because the next save would replace all your data with nothing. So it reports the problem and stops:

Error: notes.json is not valid: invalid character 'o' looking for beginning of value

Saving goes through a temporary file. os.WriteFile replaces the old content while it writes. If the program crashed partway, or the disk filled up, the real file would be left cut short. So saveNotes writes everything to notes.json.tmp first, and os.Rename then moves it over the real name, in one step. At any moment, the real file is the whole old version or the whole new version.

To be plain about what this does not do: it protects against a crash of your program in the middle of a save. It is not a full guarantee against a power cut, and it does not stop another program from writing the same file. It rewrites the whole file on every save, which gets slow with a lot of data. Real systems use a database for this. For a small program with one server, it is enough.

The number 0644 is the file's permissions, which matter on Linux and macOS. Write it as it is. On Windows it has little effect.

Notes that survive a restart

exercises/35/05-notes-saved.go puts it all together. It is 02-notes-in-memory.go with the file functions from 04-notes-file.go, and addNote now saves:

// addNote saves first, and changes the slice only if saving worked. That way the
// slice and the file never disagree.
func addNote(n note) error {
    mu.Lock()
    defer mu.Unlock()

    updated := append(slices.Clone(notes), n)
    err := saveNotes(updated)
    if err != nil {
        return err
    }
    notes = updated
    return nil
}

The file is shared too, so saving happens while the lock is held. Only one request at a time can write it, and the temporary name cannot clash. The handler turns a save failure into a status 500, because this is the server's problem and not the visitor's:

err := addNote(note{Text: text})
if err != nil {
    fmt.Println("Save error:", err)
    http.Error(w, "The note could not be saved.", http.StatusInternalServerError)
    return
}

main loads the notes before it starts the server:

func main() {
    loaded, err := loadNotes()
    if err != nil {
        fmt.Println("Could not load the notes:", err)
        fmt.Println("Fix or remove the file, then start the server again.")
        return
    }
    notes = loaded

Run it from go-work, and add two notes. Stop it, and start it again. The notes are still there, and notes.json is in your go-work folder:

[
  {
    "Text": "Buy rice"
  },
  {
    "Text": "Call Arif"
  }
]

The results when the file is damaged, with this is not JSON in it, and when it is empty:

Could not load the notes: notes.json is not valid: invalid character 'h' in literal true (expecting 'r')
Fix or remove the file, then start the server again.

Could not load the notes: notes.json is not valid: unexpected end of JSON input
Fix or remove the file, then start the server again.

Delete notes.json while the server is stopped, and it starts with an empty list.

A note containing <script> is safe in the file too. It is stored as text, and it reaches a page only through the template. encoding/json also writes < in the file as \u003c, and reads it back correctly, so do not be surprised if you open the file and see that.

Homework

Homework starters are in exercises/35/homework/. Each one runs as it is, with a placeholder. Any data file is created in the folder you run from, which is go-work. Delete it to start again.

Stuck or finished? Open the homework solutions page.