Advanced Python (Fall 2017)/lecture5

lecture 5

piping stdout to stdin

you can send the outputs (stdout) of commands in shell to other commands, or write the to files

http://www.tldp.org/LDP/abs/html/io-redirection.html

To receive the output of a pipe into your python script, use sys.stdin. sys.stdin is an iterable.

To write to stdout, you can use print or sys.stdout.write -- which does not add the newline.

To print all stdin to stdout in a python script, you could use.

import sys


def main()
    for line in sys.stdin:
        sys.stdout.write(line)


main()

and to print out a file: cat ./file.txt | python3 my_script.py

To make an stream of a file that waits for new lines to be added, use the command

tail -f ./file.txt

and you can pipe this output to your python script.

Task 1: Make a basic grep

something that can function as such tail -f ./file.txt | grep $SEARCH_TERM where lines with the search term are written to stdout

A Solution

Normal function

import argparse
import sys


def grep(term, line):
    if term in line:
        sys.stdout.write(line)


def main():

    parser = argparse.ArgumentParser()
    parser.add_argument('term')
    args = parser.parse_args()
    term = args.term
    for line in sys.stdin:
        grep(term, line)


if __name__ == '__main__':
    main()

Event driven: Using a generator

the function grep is a generator. When the file object sys.stdin yields another line, then the generator grep yields another line. When grep yeilds another line, then the for loop prints another line.

import argparse
import sys


def grep(term, lines):
    for line in lines:
        if term in line:
            print('grep executed')
            yield line


def main():

    parser = argparse.ArgumentParser()
    parser.add_argument('term')
    args = parser.parse_args()
    term = args.term
    grep_lines = grep(term, sys.stdin)
    for line in grep_lines:
        print('print executed')
        sys.stdout.write(line)


if __name__ == '__main__':
    main()

Coroutine

Please see this tutorial for a better and fuller explanation of coroutines. NOTE: this tutorial is written in Python 2 code. http://www.dabeaz.com/coroutines/

Here is our grep written with a coroutine:

import argparse
import sys


def grep(pattern):
    while True:
        line = (yield)
        if pattern in line:
            sys.stdout.write(line)


def main():

    parser = argparse.ArgumentParser()
    parser.add_argument('term')
    args = parser.parse_args()
    term = args.term
    g = grep(term)
    next(g)
    for line in sys.stdin:
        g.send(line)


if __name__ == '__main__':
    main()