35. Saving data — Homework solutions

The .go solution files are in exercises/35/homework/solutions/. Data files are created in the folder you run from. Delete them to start again.

Problem 1 — Pets in a file

Problem. Load and save a slice of structs.

How to think about it. These are the two functions from the chapter with a different type. Three outcomes for loading: no file means an empty slice and no error, an unreadable file is an error, and a file with bad JSON is an error. Saving goes through a temporary file, so that a crash cannot leave half a file.

Worked solution.

package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "os"
)

const petsFile = "pets.json"

type pet struct {
    Name string
    Age  int
}

func loadPets() ([]pet, error) {
    data, err := os.ReadFile(petsFile)
    if errors.Is(err, os.ErrNotExist) {
        return []pet{}, nil
    }
    if err != nil {
        return nil, err
    }

    var loaded []pet
    err = json.Unmarshal(data, &loaded)
    if err != nil {
        return nil, fmt.Errorf("%s is not valid: %v", petsFile, err)
    }
    return loaded, nil
}

func savePets(list []pet) error {
    data, err := json.MarshalIndent(list, "", "  ")
    if err != nil {
        return err
    }

    temporary := petsFile + ".tmp"
    err = os.WriteFile(temporary, data, 0644)
    if err != nil {
        return err
    }
    return os.Rename(temporary, petsFile)
}

func main() {
    pets, err := loadPets()
    if err != nil {
        fmt.Println("Error:", err)
        return
    }

    pets = append(pets, pet{Name: "Rex", Age: len(pets) + 1})

    err = savePets(pets)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }

    fmt.Println("Pets:", pets)
}

Two runs, then a run with a broken file:

Pets: [{Rex 1}]

Pets: [{Rex 1} {Rex 2}]

Error: pets.json is not valid: invalid character 'b' looking for beginning of value

Common mistakes.

Passing loaded to json.Unmarshal without the &. go vet reports call of Unmarshal passes non-pointer as second argument, and when run, the program prints json: Unmarshal(non-pointer []main.note) as its error, with your type in place of note.
Lower case field names. The file then holds only {}, with no error.
Checking err != nil before errors.Is. A missing file then stops the program.

Problem 2 — A visit counter

Problem. Count requests safely.

How to think about it. The count is shared and it changes, so it needs a lock. Following the rule, one small function holds the lock and does all the work, and the handler only calls it. It must read the new value inside the lock too: reading visits in the handler after the unlock could pick up another request's number.

Worked solution.

package main

import (
    "fmt"
    "net/http"
    "sync"
)

// mu protects visits.
var mu sync.Mutex
var visits = 0

func nextVisit() int {
    mu.Lock()
    defer mu.Unlock()

    visits = visits + 1
    return visits
}

func handleHome(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "You are visitor number %d.\n", nextVisit())
}

func main() {
    http.HandleFunc("GET /{$}", handleHome)

    fmt.Println("Server running at http://localhost:8080")
    fmt.Println("Press Ctrl+C to stop it.")

    err := http.ListenAndServe("localhost:8080", nil)
    if err != nil {
        fmt.Println("Server stopped:", err)
    }
}

The results:

curl.exe http://localhost:8080/
You are visitor number 1.

curl.exe http://localhost:8080/
You are visitor number 2.

The pattern "GET /{$}" matters. With "/", a browser's own request for /favicon.ico would count as a visit too, and the number would jump.

Common mistakes.

Leaving out the mutex because the counter seems to work. It does work, with one visitor. Under load, it loses counts.
Writing mu.Unlock() at the end with no defer. It works here, but it is easy to break when someone adds an early return later.
Copying the mutex, for example by passing mu as a value to a function. go vet reports passes lock by value. A copy is a different lock, and protects nothing.

Problem 3 — A counter that remembers

Problem. Keep the count in a file across restarts.

How to think about it. JSON can hold a single number as well as a list, so the file holds just 2, for example. The order inside nextVisit is the same as in the chapter: save first, and change the variable only if saving worked. That way the visitor's error and the data agree. main loads before it registers anything, and stops if the file is broken.

Worked solution.

package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "os"
    "sync"
)

const visitsFile = "visits.json"

// mu protects visits, and the file that holds it.
var mu sync.Mutex
var visits = 0

func loadVisits() (int, error) {
    data, err := os.ReadFile(visitsFile)
    if errors.Is(err, os.ErrNotExist) {
        return 0, nil
    }
    if err != nil {
        return 0, err
    }

    var loaded int
    err = json.Unmarshal(data, &loaded)
    if err != nil {
        return 0, fmt.Errorf("%s is not valid: %v", visitsFile, err)
    }
    return loaded, nil
}

func saveVisits(count int) error {
    data, err := json.Marshal(count)
    if err != nil {
        return err
    }

    temporary := visitsFile + ".tmp"
    err = os.WriteFile(temporary, data, 0644)
    if err != nil {
        return err
    }
    return os.Rename(temporary, visitsFile)
}

// nextVisit adds 1 and saves. If saving fails, the count stays as it was.
func nextVisit() (int, error) {
    mu.Lock()
    defer mu.Unlock()

    err := saveVisits(visits + 1)
    if err != nil {
        return 0, err
    }
    visits = visits + 1
    return visits, nil
}

func handleHome(w http.ResponseWriter, r *http.Request) {
    number, err := nextVisit()
    if err != nil {
        fmt.Println("Save error:", err)
        http.Error(w, "The visit could not be saved.", http.StatusInternalServerError)
        return
    }
    fmt.Fprintf(w, "You are visitor number %d.\n", number)
}

func main() {
    loaded, err := loadVisits()
    if err != nil {
        fmt.Println("Could not load the count:", err)
        fmt.Println("Fix or remove the file, then start the server again.")
        return
    }
    visits = loaded

    http.HandleFunc("GET /{$}", handleHome)

    fmt.Println("Server running at http://localhost:8080")
    fmt.Println("Press Ctrl+C to stop it.")

    err = http.ListenAndServe("localhost:8080", nil)
    if err != nil {
        fmt.Println("Server stopped:", err)
    }
}

The results, with a restart between the second and third requests:

You are visitor number 1.
You are visitor number 2.
You are visitor number 3.

And with xx in the file:

Could not load the count: visits.json is not valid: invalid character 'x' looking for beginning of value
Fix or remove the file, then start the server again.

Common mistakes.

Treating a broken file like a missing one. The count silently restarts from 0, and the next save destroys the old value.
Releasing the lock before saving. Two requests could then write the temporary file at the same time.
Adding 1 before saving, and leaving it there when saving fails. The memory and the file then disagree.

Challenge — A to-do list

Problem. Add and remove tasks, with everything from this part.

How to think about it. Plan the data functions first: allTasks, addTask and removeTask, each starting with the two lock lines. The check on n belongs inside the lock in removeTask. If the handler checked it first, another request could shorten the list between the check and the removal. That is the same kind of problem as the counter.

removeTask must tell the handler which of two things went wrong, because they get different statuses. errNoTask is a variable holding one particular error, and errors.Is checks for it, the same way as os.ErrNotExist. Anything else from removeTask is a save failure.

The list on the page is numbered by its index, and each task has a form that posts to its own address, such as /delete/2. Deleting changes data, so it is a POST and not a link.

Worked solution.

package main

import (
    "bytes"
    "encoding/json"
    "errors"
    "fmt"
    "html/template"
    "net/http"
    "os"
    "slices"
    "strconv"
    "strings"
    "sync"
    "unicode/utf8"
)

const todoFile = "todo.json"
const maxNameLength = 100

type task struct {
    Name string
}

// errNoTask means that no task has the index asked for.
var errNoTask = errors.New("no task has that index")

// mu protects tasks, and the file that holds it.
var mu sync.Mutex
var tasks = []task{}

func loadTasks() ([]task, error) {
    data, err := os.ReadFile(todoFile)
    if errors.Is(err, os.ErrNotExist) {
        return []task{}, nil
    }
    if err != nil {
        return nil, err
    }

    var loaded []task
    err = json.Unmarshal(data, &loaded)
    if err != nil {
        return nil, fmt.Errorf("%s is not valid: %v", todoFile, err)
    }
    return loaded, nil
}

func saveTasks(list []task) error {
    data, err := json.MarshalIndent(list, "", "  ")
    if err != nil {
        return err
    }

    temporary := todoFile + ".tmp"
    err = os.WriteFile(temporary, data, 0644)
    if err != nil {
        return err
    }
    return os.Rename(temporary, todoFile)
}

func allTasks() []task {
    mu.Lock()
    defer mu.Unlock()

    return slices.Clone(tasks)
}

func addTask(t task) error {
    mu.Lock()
    defer mu.Unlock()

    updated := append(slices.Clone(tasks), t)
    err := saveTasks(updated)
    if err != nil {
        return err
    }
    tasks = updated
    return nil
}

// removeTask removes the task at index n, counting from 0. The check is inside
// the lock, because another request could change the list at any moment.
func removeTask(n int) error {
    mu.Lock()
    defer mu.Unlock()

    if n < 0 || n >= len(tasks) {
        return errNoTask
    }

    updated := slices.Clone(tasks)
    updated = append(updated[:n], updated[n+1:]...)

    err := saveTasks(updated)
    if err != nil {
        return err
    }
    tasks = updated
    return nil
}

const todoText = `<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>To-do</title>
</head>
<body>
<h1>To-do</h1>
<ul>
{{range $i, $task := .}}
<li>
{{$task.Name}}
<form action="/delete/{{$i}}" method="post">
<button type="submit">Delete</button>
</form>
</li>
{{else}}
<li>Nothing to do.</li>
{{end}}
</ul>
<form action="/tasks" method="post">
<p><label>New task: <input type="text" name="name"></label></p>
<p><button type="submit">Add</button></p>
</form>
</body>
</html>
`

var todoTemplate = template.Must(template.New("todo").Parse(todoText))

func handleHome(w http.ResponseWriter, r *http.Request) {
    var page bytes.Buffer
    err := todoTemplate.Execute(&page, allTasks())
    if err != nil {
        fmt.Println("Template error:", err)
        http.Error(w, "The page could not be built.", http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "text/html; charset=utf-8")
    page.WriteTo(w)
}

func handleAdd(w http.ResponseWriter, r *http.Request) {
    name := strings.TrimSpace(r.FormValue("name"))
    if name == "" {
        http.Error(w, "The task has no name.", http.StatusBadRequest)
        return
    }
    if utf8.RuneCountInString(name) > maxNameLength {
        http.Error(w, "The name is too long.", http.StatusBadRequest)
        return
    }

    err := addTask(task{Name: name})
    if err != nil {
        fmt.Println("Save error:", err)
        http.Error(w, "The task could not be saved.", http.StatusInternalServerError)
        return
    }

    http.Redirect(w, r, "/", http.StatusSeeOther)
}

func handleDelete(w http.ResponseWriter, r *http.Request) {
    n, err := strconv.Atoi(r.PathValue("n"))
    if err != nil {
        http.Error(w, "The index must be a whole number.", http.StatusBadRequest)
        return
    }

    err = removeTask(n)
    if errors.Is(err, errNoTask) {
        http.Error(w, err.Error(), http.StatusNotFound)
        return
    }
    if err != nil {
        fmt.Println("Save error:", err)
        http.Error(w, "The list could not be saved.", http.StatusInternalServerError)
        return
    }

    http.Redirect(w, r, "/", http.StatusSeeOther)
}

func main() {
    loaded, err := loadTasks()
    if err != nil {
        fmt.Println("Could not load the list:", err)
        fmt.Println("Fix or remove the file, then start the server again.")
        return
    }
    tasks = loaded

    http.HandleFunc("GET /{$}", handleHome)
    http.HandleFunc("POST /tasks", handleAdd)
    http.HandleFunc("POST /delete/{n}", handleDelete)

    fmt.Println("Server running at http://localhost:8080")
    fmt.Println("Press Ctrl+C to stop it.")

    err = http.ListenAndServe("localhost:8080", nil)
    if err != nil {
        fmt.Println("Server stopped:", err)
    }
}

The results, after adding three tasks and deleting the first:

curl.exe -i -X POST http://localhost:8080/delete/0
HTTP/1.1 303 See Other
Location: /

curl.exe -i -X POST http://localhost:8080/delete/99
HTTP/1.1 404 Not Found
no task has that index

curl.exe -i -X POST http://localhost:8080/delete/abc
HTTP/1.1 400 Bad Request
The index must be a whole number.

curl.exe -i -d "name=" http://localhost:8080/tasks
HTTP/1.1 400 Bad Request
The task has no name.

curl.exe -i http://localhost:8080/delete/0
HTTP/1.1 405 Method Not Allowed
Allow: POST

Method Not Allowed

Task names with <script> in them show as text, because the template escapes them.

Common mistakes.

Making delete a GET link. A browser or another program may fetch a link by itself, and a task disappears.
Reading allTasks() in the handler to check n, then calling removeTask. The list can change between the two calls.
Calling allTasks() from inside addTask. It waits for a lock that its own caller holds, and the server hangs on that request.
Forgetting that a task's index changes when an earlier task is deleted. A page left open shows old numbers, so a stale form can delete a different task. That is an acceptable limit here. A real program gives each item a fixed id.