Python's Generator and Yield Explained

What are generators?

Generators are iterators, a kind of iterable you can only iterate over once.

What are iterators?

An iterator is an object that can be iterated (looped) upon. It is used to abstract a container of data to make it behave like an iterable object. Some common iterable objects in Python are - lists, strings, dictionary.

Every generator is an iterator, but not vice versa. A generator is built by calling a function that has one or more yield expressions.

The yield keyword behaves like return in the sense that values that are yielded get “returned” by the generator. Unlike return, the next time the generator gets asked for a value, the generator’s function, resumes where it left off after the last yield statement and continues to run until it hits another yield statement.

In simpler words, a generator is simply a function that returns a generator object on which you can call next() such that for every call it returns some value until it raises a StopIteration exception, signaling that all values have been generated.

Let’s take some examples of generators:

def generator_example():
    yield "Iterator first object"
    yield "Iterator second object"
    yield "Iterator third object"

As per the definition, the generator function creates a generator object you can verify this by just calling the function..

$ generator_example()

Output:

<generator object generator_example at 0x7f0980615348>

To get the actual result just, Store the object in a variable and call the next() method on it. Every call on next() will yield a single value until all the values have been yield.

gen_obj = generator_example()
next(gen_obj)

Output:

Iterator first object

Again from the definition, every call to next will return a value until it raises a StopIteration exception, signaling that all values have been generated so for this example we can call the next method 3 times since there are only 3 yield statements to run.

2nd Iteration

next(gen_obj)

Output:

Iterator second object

3rd Iteration

next(gen_obj)

Output:

Iterator third object

If you call next(gen_obj) for the fourth time, you will get StopIteration error from the Python interpreter.

4th Iteration

next(gen_obj)

Output:

Traceback (most recent call last):
...
StopIteration

Note: We can use generators with for loops directly, because a for loop takes an iterator and iterates over it using next() function. It automatically ends when StopIteration is raised.

for item in gen_obj:
    print(item)

Output:

Iterator first object
Iterator second object
Iterator third object

Let’s take an another example where we will generator random integer number between 0 to 500 and then will get using the generators.

import random

def random_number_generator():
    while True:
        number = random.randint(0, 500)
        yield number

Here the generator function will keep returning a random number since there is no exit condition from the loop.

num = random_number_generator()
next(num)

Output:

134

Generator Expression

Simple generators can be easily created on the fly using generator expressions. It makes building generators easy. The syntax for generator expression is similar to that of a list comprehension in Python. But the square brackets[] are replaced with round parentheses().

What is the difference between list comprehension and generator expression?

• The major difference between a list comprehension and a generator expression is that a list comprehension produces the entire list while the generator expression produces one item at a time.
• They have lazy execution ( producing items only when asked for ).

For this reason, a generator expression is much more memory efficient than an equivalent list comprehension.

Example:

# Initialize the list
my_list = [1, 7, 2, 11]

# square each term using list comprehension
list_squares = [x**2 for x in my_list]

# same thing can be done using a generator expression
# generator expressions are surrounded by parenthesis ()
gen = (x**2 for x in my_list)

print(list_squares)
print(gen)

Output

[1, 7, 2, 11]
<generator object <genexpr> at 0x8f4d3eb4cf70>

You can use next() to start getting output or you can also use for loop to iterator the output.

next(gen)

If you can to get generato ouput in list or tuple then you can directly convert generator object into the list or tuple.

list(gen)
tuple(gen)

Done!!

When are generators useful?

The key advantage to use the generators is that the “state” of the function is preserved, unlike with regular functions where each time the stack frame is discarded, you lose all that “state”. Also, generators do not store all the values in memory instead they generate the values on the fly thus making the ram more memory efficient.

Conclusion
Generator functions are ordinary functions defined using yield instead of return. When called, a generator function returns a generator object, which is a kind of iterator - it has a next() method. When you call next(), the next value yielded by the generator function is returned.