Process text into data 🔥

A puzzle hands you text and wants answers in numbers. Before you can solve the interesting part, you often need to clean that text, pull out the pieces you need, and turn them into values your program can work with. Real input also has an annoying habit of containing spaces, units, missing values, and other surprises.

In this chapter, you’ll return to the sensor log from Work with files and Work with strings. You’ll split and clean its lines, extract individual fields, convert temperatures to Float64, and collect the results in a typed list. Then you’ll deliberately add a bad reading and make your parser deal with it, so one broken line doesn’t stop you from processing everything else.

Splitting lines

Create parse_text.mojo, and split the log into separate lines, the same way you did in Work with files:

    var log = Path("src/downloads/temp_log.txt")
    var text = log.read_text()
    var lines = text.splitlines()
    print(t"Got {len(lines)} lines")  # Got 10 lines

Try this

Replace splitlines() with split("\n"):

    var split_on_newline = text.split("\n")
    print(t"split got {len(split_on_newline)} lines")  # 11 lines

The final newline becomes an extra empty element: 11 entries instead of 10. For line-oriented text, prefer splitlines().

Tidy your input

Some of Chiller’s lines carry stray leading or trailing whitespace. Strip each line, skip anything blank, and save the result in cleaned_lines so every step from here on works with tidy data instead of stripping again each time:

    var cleaned_lines: List[String] = []
    for line in lines:
        var cleaned = line.strip()
        if not cleaned:
            continue
        cleaned_lines.append(String(cleaned))
        print(t"line: '{cleaned}'")

Takeaways

strip() returns a StringSpan with leading and trailing whitespace removed.
Empty strings and string slices are false in conditional expressions.
continue skips the rest of the current iteration.
Wrapping each cleaned slice in String(...) copies it out of text, so cleaned_lines stays valid on its own.

Extract the fields

Each line packs three pieces of information into one string: the day, the temperature, and the conditions. Split on the same separators the log uses to pull them apart:

    var first = cleaned_lines[0]
    var reading = first.split(": ")[1].split(", ")
    var temp_field = reading[0]  # '-20.5C'
    var conditions = reading[1]  # 'Partly Cloudy'
    print(t"temp field: '{temp_field}', conditions: '{conditions}'")

Takeaways

split(sep) returns a List[StringSpan], a view into the original text.
Chaining split() calls peels off one layer of structure at a time.

Type conversion

The temperature field still carries its unit. Try converting it directly:

    try:
        _ = Float64(temp_field)
    except e:
        print(t"Float64('{temp_field}') raised: {e}")

Float64() raises. The trailing C isn’t part of the number.

Strip the unit first:

    var temp = Float64(temp_field.removesuffix("C"))
    print(t"temp: {temp}")  # -20.5

Takeaways

Float64(value) raises when the text isn’t a valid floating-point value.
removesuffix(suffix) strips the substring only if it appears at the end, and returns the text unchanged otherwise.

Collect all readings

Put the extraction and conversion together for every line in cleaned_lines:

    var temps: List[Float64] = []

    for line in cleaned_lines:
        var fields = line.split(": ")[1].split(", ")
        temps.append(Float64(fields[0].removesuffix("C")))

    print(t"Parsed {len(temps)} temperatures: {temps}")

Ten parsed readings.

Takeaways

append() adds a value to the end of a list.
Typed lists like List[Float64] catch mismatched values at compile time.

Handle invalid lines

Sensors fail. Copy cleaned_lines and simulate one more reading coming in corrupted:

    # Simulate one more reading coming in from a flaky sensor
    var extended_lines = cleaned_lines.copy()
    extended_lines.append("Day 11: ERROR, Sensor Fault")

Catch conversion errors and keep parsing instead of crashing:

    temps = []
    var rejected: List[String] = []

    for line in extended_lines:
        var fields = line.split(": ")[1].split(", ")

        try:
            temps.append(Float64(fields[0].removesuffix("C")))
        except:
            rejected.append(line)

    print(t"Parsed {len(temps)} temperatures: {temps}")
    print(t"Rejected {len(rejected)}: {rejected}")

Ten parsed readings and one rejected value.

Takeaways

Catch a conversion error to reject one bad reading and continue parsing the rest.
The rejected list stores String copies rather than views into the original line.

Try this

except: catches and discards the error. Use except e: to inspect the error.

Change except to except e and then print the error with the rejected line:

    for line in extended_lines:
        var fields = line.split(": ")[1].split(", ")

        try:
            _ = Float64(fields[0].removesuffix("C"))
        except e:
            print(t"Rejected '{line}': {e}")

Final code

Your complete parse_text.mojo:

from std.pathlib import Path


def main() raises:
    var log = Path("src/downloads/temp_log.txt")
    var text = log.read_text()
    var lines = text.splitlines()
    print(t"Got {len(lines)} lines")  # Got 10 lines

    var split_on_newline = text.split("\n")
    print(t"split got {len(split_on_newline)} lines")  # 11 lines

    var cleaned_lines: List[String] = []
    for line in lines:
        var cleaned = line.strip()
        if not cleaned:
            continue
        cleaned_lines.append(String(cleaned))
        print(t"line: '{cleaned}'")

    var first = cleaned_lines[0]
    var reading = first.split(": ")[1].split(", ")
    var temp_field = reading[0]  # '-20.5C'
    var conditions = reading[1]  # 'Partly Cloudy'
    print(t"temp field: '{temp_field}', conditions: '{conditions}'")

    try:
        _ = Float64(temp_field)
    except e:
        print(t"Float64('{temp_field}') raised: {e}")

    var temp = Float64(temp_field.removesuffix("C"))
    print(t"temp: {temp}")  # -20.5

    var temps: List[Float64] = []

    for line in cleaned_lines:
        var fields = line.split(": ")[1].split(", ")
        temps.append(Float64(fields[0].removesuffix("C")))

    print(t"Parsed {len(temps)} temperatures: {temps}")

    # Simulate one more reading coming in from a flaky sensor
    var extended_lines = cleaned_lines.copy()
    extended_lines.append("Day 11: ERROR, Sensor Fault")

    temps = []
    var rejected: List[String] = []

    for line in extended_lines:
        var fields = line.split(": ")[1].split(", ")

        try:
            temps.append(Float64(fields[0].removesuffix("C")))
        except:
            rejected.append(line)

    print(t"Parsed {len(temps)} temperatures: {temps}")
    print(t"Rejected {len(rejected)}: {rejected}")

    for line in extended_lines:
        var fields = line.split(": ")[1].split(", ")

        try:
            _ = Float64(fields[0].removesuffix("C"))
        except e:
            print(t"Rejected '{line}': {e}")

    var row = cleaned_lines[5]  # 'Day 6: -27.7C,   Foggy'
    var columns = row.split(",")
    print(columns)  # [Day 6: -27.7C,    Foggy]

Topics covered

Splitting text, stripping whitespace, chained field extraction, string truthiness, continue, typed lists, append(), conversion to Float64, removesuffix(), and error handling.

Keyboard shortcuts

Advent of Mojo