Generator Functions in Python
Tags: python, functional, programming, programmers
Published: 2024-03-20 22:37:40

<p>Generators are special functions in which execution does not happen all at once like in traditional functions. Instead,&nbsp;&nbsp;generators can pause the execution and resume later from the same point.&nbsp;&nbsp;</p>

<p>By definition, generators&nbsp;are functions that contains&nbsp;one or more&nbsp;<code>yield</code> statements. The <code>yield </code>statement is similar to the <code>return </code>statement except that when it is encountered,&nbsp; it does not immediately cause the function to exit&nbsp;. Instead, it causes the function to suspend its execution and returns the given value(if any) to the caller. The execution can then be resumed by calling the <code>next()</code> function on the generator.</p>

<p>Consider the following example:</p>

<div class="autorun example">
<pre>
<code class="language-python">def func():
    print('Hello, World!')
    yield 1
    print('You are at Pynerds')
    yield 2
    print('Enjoy Your stay')
    yield 3

gen = func()
print(next(gen))
print('....................')
print(next(gen))
print('....................')
print(next(gen))</code></pre>
</div>

<p>As shown above, when a generator function is called&nbsp;it returns a generator object which is a typical iterator. Being an iterator, the object supports the iterator protocol, which means that it can be used with the <code>next() </code>function and other iteration constructs such as for loops.</p>

<p>The <code>next()</code>&nbsp;function makes the generator to start or resume its execution until another yield statement is encountered or the execution is complete. We can also use the generator with a for loop, which automatically&nbsp;calls the <code>next()</code> function on the generator until it reaches the end of the execution.&nbsp;</p>

<div class="autorun example">
<pre>
<code class="language-python">def func():
   L = [10, 20, 30, 40, 50, 60]
   for i in L:
       yield i

for i in func():
    print(i)</code></pre>
</div>

<p>As shown in the above example, the <code>yield</code> statement&nbsp; can be used anywhere in the function&#39;s body like in loops, condition blocks, etc.</p>

<h2>How generators work</h2>

<p>Consider what happens when we call a normal function i.e those using the&nbsp;<code>return&nbsp;</code>statement. Whenever the <code>return </code>statement is encountered, the function simply returns the value and terminates without retaining any information on its internal state, any subsequent calls to the function will start fresh and execute the same code again.</p>

<p>On the other hand, whenever a generator object encounters a <code>yield </code>statement, it temporarily suspends/pauses the function and saves its state. This allows the generator to &quot;remember&quot; where it left off when it is called again, and it can resume execution from that point on rather than&nbsp; starting all over again.&nbsp;This makes generators very efficient when dealing with large datasets or long-running calculations, since it allows you to get than results in chunks rather than&nbsp;loading the entire dataset into memory or waiting for a long-running calculation to finish before returning the results.</p>

<div class="autorun example">
<pre>
<code class="language-python">def my_gen():
   data = range(10)
   print("started")
   for i in data:
       if i == 5:
           print('We are halfway.')
       yield i
   print('Done')

gen = my_gen()
for i in gen:
    print(i)</code></pre>
</div>

<h2>Generator with a return statement</h2>

<p>When a <code>return </code>statement is encountered in a generator object, the&nbsp; generator gets closed and any subsequent calls will result in&nbsp;a <code>StopIteration </code>exception.</p>

<div class="autorun example">
<pre>
<code class="language-python">def func():
   data = range(10)
   for i in data:
       yield i
       if i == 5:
           return

for i in func():
    print(i)</code></pre>
</div>

<h2>The next function</h2>

<p>The builtin &nbsp;<a href="https://www.pynerds.com/python-next-function/">next()</a> function is a useful feature when working with iterators. It is used to retrieve the next item from an iterator object. The function raises a <code>StopIteration </code>exception if the iterator is empty.</p>

<p>When called on a generator object, the <code>next() </code>function&nbsp;makes the&nbsp;the generator to&nbsp;proceed with the execution from where it had stopped until the next <code>yield </code>statement is reached or the execution is done.</p>

<div class="autorun example">
<pre>
<code class="language-python">def func():
   L = range(10)
   for i in L:
       yield i

gen = func()
L = list(gen)

print(*L)

#The generator object is now empty
next(gen)</code></pre>
</div>

<p>As shown above, trying to access the next value in an empty generator object&nbsp;will raise a&nbsp;<code>StopIteration</code> exception. We can use the<code> try</code> blocks&nbsp;to catch the<code>&nbsp;StopIteration&nbsp;</code>exception. Example:</p>

<div class="autorun example">
<pre>
<code class="language-python">def func():
   L = [1, 4, 8, 9]
   for i in L: 
      yield i

gen = func()
try:
    while True:
        value = next(gen)
        print(value)
except StopIteration:
    print("Nothing else")</code></pre>
</div>

<h2>Closing the generator&nbsp;</h2>

<p>The generator objects contains the close() method which can be used to pre-maturely&nbsp;close the generator and free up any resources being used.</p>

<div class="autorun example">
<pre>
<code class="language-python">def func():
    data = ['Python', 'Java', 'C++', 'Ruby', 'Swift']
    for i in data:
        yield i

gen = func()
print(next(gen))
print(next(gen))
print(next(gen))

#close the generator
gen.close()

#subsuent calls will raise a StopIteration exception
next(gen)</code></pre>
</div>

<h2>Controlling the generator from outside</h2>

<p>When we are performing subsequent calls to the generator, we can use the <code>send()</code> method to send a value to the generator. This method can be thought partly as the opposite&nbsp;of the <code>yield </code>statement. It resumes&nbsp;the function execution and receives the yielded value&nbsp;just like the <code>next()</code> function, but also sends an external value back to the <code>yield </code>statement.</p>

<div class="syntax">
<pre>
<code class="language-python">send(obj)</code></pre>
</div>

<p>The object sent is received by the <code>yield </code>statement.&nbsp; We can only send a single object.</p>

<div class="autorun example">
<pre>
<code class="language-python">def square():
    recieved = None
    while True:
        if recieved is None: 
            recieved = yield "No value given."
        else:
            recieved = yield f"{recieved} ^ 2 = {recieved ** 2}"

gen = square()
print(gen.send(None))
print(gen.send(5))
print(gen.send(10))
print(gen.send(21))
print(gen.send(25))
print(gen.send(30))
print(gen.send(50))

gen.close()</code></pre>
</div>

<p>Note that we cannot send a value(other than <code>None</code>) in the first yield call.</p>

<h2>Get the Internal state of the generator object</h2>

<p>Generator objects contains some attributes which we can use to get its internal state. For example the <code>gi_suspended</code> and the <code>gi_running </code>attributes returns a boolean value indicating whether the generator object is currently paused&nbsp;or running, respectively.</p>

<pre>
<code class="language-python">def func():
    data = [1, 2, 3, 4, 5]
    for i in data:
        yield i

gen = func()

print(next(gen))

print('running: ', gen.gi_running)
print('suspended: ', gen.gi_suspended)</code></pre>

<div class="output">
<p>1<br />
running: &nbsp;False<br />
suspended: &nbsp;True&nbsp;</p>
</div>

<h2>Lazy Evaluation in Generator Functions</h2>

<p>Lazy evaluation is an optimization technique in which&nbsp;the evaluation of an expression is delayed until the value is needed.&nbsp;Generator functions use&nbsp;this technique such that the values returned are not loaded in memory all at once, instead, the items are efficiently processed one at a time as needed.This makes them capable of efficiently handling large data sets which would otherwise take up a lot of memory space.</p>




<div class="tut-index col-lg-2 col-md-3" id="tut-index"><h4 class="text-black">Python Functions</h4><a href="https://www.pynerds.com/introduction-to-functions-in-python/" class="text-dark"><p class="left-item">Introduction</p></a><a href="https://www.pynerds.com/python-functions/" class="text-dark"><p class="left-item">functions</p></a><a href="https://www.pynerds.com/python-function-scope/" class="text-dark"><p class="left-item">function scope</p></a><a href="https://www.pynerds.com/return-multiple-values-in-python-functions/" class="text-dark"><p class="left-item">Multiple return Values</p></a><a href="https://www.pynerds.com/python-lambda-functions/" class="text-dark"><p class="left-item">lambda functions</p></a><a href="https://www.pynerds.com/generator-functions-in-python/" class="text-dark"><p class="bg-default text-dark left-item">Generator Functions</p></a><a href="https://www.pynerds.com/recursion-in-python/" class="text-dark"><p class="left-item">recursion</p></a><a href="https://www.pynerds.com/function-decorators-in-python/" class="text-dark"><p class="left-item">Decorators</p></a><a href="https://www.pynerds.com/partial-functions-in-python/" class="text-dark"><p class="left-item">partial functions</p></a><a href="https://www.pynerds.com/docstrings-in-python/" class="text-dark"><p class="left-item">Function docstring</p></a></div>