Python Tips: Generator unrolling

Generators are computed iterables, which only require a fraction of the space that would normally be necessary to store a fully populated collection in memory.

While this property is generally beneficial, they also bring along some ergonomic problems:

The most common way to deal with these shortcomings is to just unroll the generator into a collection if the element count yielded by the generator is manageable. There are many ways to accomplish this, but all of them have different strengths and shortcomings.

Benchmark code
from timeit import timeit
from functools import partial
from array import array
from sys import getsizeof

RUNS = 1000
RANGE = 10000

for exprs, desc in (
    (lambda gen: list(gen), "List constructor"),
        lambda gen: (lambda gen, list: [list.append(num) for num in gen])(gen, list()),
        "List append",
    (lambda gen: [num for num in gen], "List comprehension"),
    (lambda gen: tuple(gen), "Tuple constructor"),
    (lambda gen: set(gen), "Set constructor"),
    (lambda gen: frozenset(gen), "Frozenset constructor"),
    (lambda gen: {num: None for num in gen}, "Dictionary nonsense"),
    (lambda gen: array("h", gen), "Array constructor"),
        lambda gen: (lambda gen, arr: arr.extend(gen) or arr)(gen, array("h")),
        "Array extension",
        lambda gen: (lambda gen, arr: [arr.append(num) for num in gen])(
            gen, array("h")
        "Array append",
    bound_expr = partial(exprs, (lambda: range(RANGE))())
    print(f"{timeit(bound_expr, number=RUNS):.5f}s: {desc} ({getsizeof(bound_expr())})")
MethodDescriptionSpeed (sec)Size (byte)
list(gen)List constructor0.2717590112
tuple(gen)Tuple constructor0.3514580048
frozenset(gen)Frozenset constructor0.47581524512
set(gen)Set constructor0.47707524512
[num for num in gen]List comprehension0.5215587624
{num: None for num in gen}Dictionary nonsense0.74752295008
array("h", gen)Array constructor (from array import array)0.9089520234
arr.extend(gen)Array extension0.9403720234
[list.append(num) for num in gen]List append1.1985587624
[arr.append(num) for num in gen]Array append1.7788287624


Attribution-NonCommercial 4.0 International (only applies to text, code license: MIT)