Restarting a Python Script Within Itself

By Petr Zemek in Software Engineering 29 Comments

Sometimes, you may wish to check within a script when a configuration file or the script itself changes, and if so, then automatically restart the script. In this post, you will see a way of doing this in Python.

Consider the following scenario. You have a Python script that runs as a daemon and regularly performs the prescribed tasks. Examples may be a web server, a logging service, and a system monitor. When the script starts, it reads its configuration from a file, and then enters an infinite loop. In this loop, it waits for inputs from the environment and acts upon them. For example, a web server may react to a request for a page, which results into sending a response to the user.

From time to time, it may be necessary to restart the script. For example, if you fix a bug in it or change its configuration. One way of doing so is to kill the script and run it again. However, this requires manual intervention, which you may forget to do. When you fix a vulnerability in the script, you want to be sure that you do not forget to restart the script. Otherwise, someone may exploit the vulnerability if you did not restart the script. It would be nice if there existed a way of restarting the script within itself after it detected that its sources or a configuration file changed. In the rest of this post, we will show such a way.

For the purpose of the present post, let us assume that the script has the following structure:

# Parse the arguments and configuration files.
 
while True:
    # Wait for inputs and act on them.
    # ...

That is, it processes the arguments and loads the configuration from the configuration files. After that, the script waits for inputs and processes them in an infinite loop.

Next, we describe how to watch files for changes. After that, we show how to restart the script.

Checking Watched Files For Changes

First, we define the paths to the files whose change we want to watch:

WATCHED_FILES = [GLOBAL_CONFIG_FILE_PATH, LOCAL_CONFIG_FILE_PATH, __file__]

We watch the global configuration file, the local configuration file, and the script itself, whose path can be obtained from the special global variable __file__. When the script starts, we get and store the time of the last modification of these files by using os.path.getmtime():

from os.path import getmtime
 
WATCHED_FILES_MTIMES = [(f, getmtime(f)) for f in WATCHED_FILES]

Then, we add a check if any of these files have changed into the main loop:

while True:
    for f, mtime in WATCHED_FILES_MTIMES:
        if getmtime(f) != mtime:
            # Restart the script.
 
    # Wait for inputs and act on them.
    # ...

If either of the files that we watch has changed, we restart the script. The restarting is described next.

Restarting the Script

We restart the script by utilizing one of the exec*() functions from the os module. The exact version and arguments depend on how you run the script. For example, on Linux or Mac OS, you can make the file executable by putting the following line to the top of the file

#!/usr/bin/env python

and executing

$ chmod a+x daemon.py

Then, you can run the script via

$ ./daemon.py

In such a situation, to restart the script, use the following code:

os.execv(__file__, sys.argv)

Otherwise, when you run the script via

$ python daemon.py

use this code:

os.execv(sys.executable, ['python'] + sys.argv)

Either way, do not forget to import the sys module:

import sys

To explain, the arguments of os.execv() are the program to replace the current process with and arguments to this program. The __file__ variable holds a path to the script, sys.argv are arguments that were passed to the script, and sys.executable is a path to the Python executable that was used to run the script.

The os.execv() function does not return. Instead, it starts executing the current script from its beginning, which is what we want.

Concluding Remarks

If you use the solution above, please bear in mind that the exec*() functions cause the current process to be replaced immediately, without flushing opened file objects. Therefore, if you have any opened files at the time of restarting the script, you should flush them using f.flush() or os.fsync(fd) before calling an exec*() function.

Of course, the presented solution is only one of the possible ways of restarting a Python script. Depending on the actual situation, other approaches, like killing the script externally and starting it afterwards, may be more suitable for you. Moreover, there exist other methods of checking whether a watched file has changed and acting upon such a change. If you know of another way of restarting a Python program within itself, please share it by posting a comment.

Complete Source Code

The complete source code for this post is available on GitHub.

29 Comments

  1. Petr,

    This looks to be exactly what I am looking for.
    I am very new to Python though so I am struggling figuring out how exactly to implement this.
    I have a cron job that runs (lets call it program A) every night at midnight and goes to a webservice and checks to see if i have updated my code to my program (lets call it program B). If I have, it downloads the code and overwrites my program B code. Until know I couldn’t get program A to restart program B. So I set up a cron job that just reboots the PI every night. I don’t like this and would really like Program B to pick up the change and restart itself only if there has been a change. I am struggling on where to put this while loop in my code.

    lets say i have a stop watch program that essentially shows a clock and a stop watch at the same time. Its always waiting on external inputs from GPIO to start and stop the stop watch and record their times. I tried putting your loop at the top and the bottom but my code either goes into an infinite loop or doesn’t display the main program. Any advice you could give would be appreciated.

    def main():
        root = Tk()
        FS = FullScreenApp(root)
        cf = ConfigFile(root)
        frame = Frame(root)
        app = App(root)
        sw = StopWatch(root)
        root.mainloop()
    Reply

    • Hi Justin!

      You have to put the checking and restarting into a place that is periodically executed. For example, in a Tk application, this can be done by using root.after() as follows:

      #!/usr/bin/env python
       
      import os
      import sys
      # If you use Python 2, replace tkinter with Tkinter.
      from tkinter import Tk
       
      # Number of milliseconds after which a check whether
      # the file has changed is performed.
      CHECK_INTERVAL = 2000
       
      # Last time the file was modified.
      last_mtime = os.path.getmtime(__file__)
       
      # The main window.
      root = Tk()
       
      def restart_if_changed():
          # Check if the file has changed.
          # If so, restart the application.
          if os.path.getmtime(__file__) > last_mtime:
              print('file has changed => restarting')
       
              # Restart the application (os.execv() does not return).
              os.execv(__file__, sys.argv)
       
          # Reschedule the checking.
          root.after(CHECK_INTERVAL, restart_if_changed)
       
      # Schedule the checking.
      root.after(CHECK_INTERVAL, restart_if_changed)
       
      # Enter the main processing loop.
      root.mainloop()

      Just save the code into a file, make the file executable, and run it. Then, whenever you modify the file (its mtime attribute is changed), the application is automatically restarted. You can try it by yourself: run the application, modify the file in a text editor, and after at most two seconds, the application should be restarted.

      Reply

      • But what if I accidentally introduce an error in the new version of this .py file? I would want the old version to keep on running (and output the error), and *try* restarting again when the file is next updated.

        Your solution would have to be manually restarted after the problem is fixed.

        Reply

  2. Hello Petr

    I’m running a script on a beaglebone black with debian
    Here is the part that gets the error.

    for f, mtime in watched_files_mtimes:
      if getmtime(f) > mtime:
      writeProgLog("File change")
      print "Restarting program! -- file changed"
      os.execv(__file__, sys.argv)
    threading.Timer(20, every20Seconds).start()

    this is the error

        os.execv(__file__, sys.argv)
    OSError: [Errno 13] Permission denied

    PS. I guess I need to figure out how to post a code block
    thanks
    bill

    Reply

      • Hi. Once I make the python file executable I got a new error:

        Traceback (most recent call last):
          File "file.py", line 30, in
            os.execv(__file__, sys.argv)
        OSError: [Errno 8] Exec format error

        I’ve checked and tried some shebangs but nothing works.

        I don’t know what os.execv(__file__, sys.argv) exacly does, but seems a popen command at all.
        This is what i did to relaunch the script:

        def relaunch():
            args = "" # must declare first
            for i in sys.argv:
                args = args + " " + str(i) # force casting to strings with str() to concatenate
            print "python" + args # print what would we run
            os.system("python" + args) # do it!

        Python script calling relaunch will launch itself again. I’m sure this solution do not cover memory handling or other aspects, but I need the script reloading at any cost :D

        Reply

        • Hi erm3nda. To fix the Exec format error, put the following line to the top of the file:

          #!/usr/bin/env python

          This line informs the system to run the script in Python. When this line is not present, the system does not know how to execute the script when it is relaunched.

          Reply

    • An easy way to restart the Python Shell programmatically is as follows

      import _winapi
      x = _winapi.GetCurrentProcess()
      def RESTART() : _winapi.ExitProcess(x)
      RESTART()

      That was one way I found.

      Reply

  3. @Petr Zemek about “#!/usr/bin/env python”

    I saw that I tried some shebangs and still doesn’t work.
    This time im working on a Windows machine and shebang does exactly nothing on it.

    I’ve checked .py assoc with pythonw.exe instead python.exe and still doesn’t work.
    I’m really stucked with that because that would be dead simple and is not.

    Another way would be to create a simple Launcher just for that.

    Reply

  4. Nice code, just what I was looking for. Thanks!
    What if the new script does eventually not work as expected and you revert back to the backed-up original? In that case

    if getmtime(f) > mtime:

    prevents the restart because the backup file has a mtime that is < than the mtime of the running script.
    I would say: use

    if getmtime(f) <> mtime:
    Reply

  5. Hi Petr

    1st of all, let me thank you very much for your explanation. I inserted your code fragment into a timer. There are two routines: 1 routine compares the NOW-time to the saved times in a list and switches the respective GPIO, when NOW- and SAVE-time are the same. This routine runs Daemon-like in the background and is started by a cronjob @reboot. The other routine let’s me change the saved times (switch times) and the names of the channels. Now, when I’m changing times or names, I save the new list with ‘pickle’ to a file. When this file is changed, the switch-routine restarts itself and reloads the pickle file to adopt the new settings. I know, it’s surely not the fine English style of coding, but it works flawlessly, that’s the essential.

    2 things to remark: First of all, your code runs flawlessly unter Python 3 as well. Just use

    #!/usr/bin/python3

    as interpreter description (I assume it is installed…). Secondly, you can check every single file you want. Just use

    WATCHED_FILES = ['/home/user/.../nameofthescript.py']

    Just for the beginners who don’t know – like me. (Trial and error – method…) :)

    Reply

    • I also got the same problem.
      It seems that __file__ refers to the compiled bytecode file with .pyc instead of the original .py.
      Does this break the os.exec statement when trying to exec the bytecode directly?

      Reply

  6. Here’s my solution. It can relaunch the file even after you introduced errors.

    # usage:
    #   RelaunchWatchdog.py <subfile>.py
     
    # This program takes a .py file which it immediately executes in a subprocess.
    # It kills the subprocess and relaunches it whenever the file changes.
    #
    # This program is independent of the python file it launches:
    # If you accidentally introduce an error into the .py file, just fix that error and save it again, and it will be relaunched.
    # Note that the subprocess shared the console with this process, so you will see its output.
     
    import sys,time,subprocess,logging, os, sys, watchdog, watchdog.events, watchdog.observers,psutil
     
    subfile = sys.argv[1]
     
    def launch():
      global process
      process = subprocess.Popen(subfile, shell=True)
     
    def samefile(a,b):
      return os.stat(a) == os.stat(b)
     
    def kill_or_do_nothing():
      global process
      try: # Process might already have terminated
        process = psutil.Process(process.pid)
        for proc in process.get_children(recursive=True):
          proc.kill()
        process.kill()
      except:
        pass
         
    class Modified(watchdog.events.FileSystemEventHandler):
        def on_modified(self, event):
            if not samefile(event.src_path, subfile):
                return
             
            print subfile, 'modified, will restart'
            kill_or_do_nothing()
            launch()
     
    observer = watchdog.observers.Observer()
    observer.schedule(Modified(), '.')
    observer.start()
     
    launch()
     
    # Idle forever, listening on Modified.on_modified
    while True:
      time.sleep(1)
    Reply

  7. man i will take back this

    Traceback (most recent call last):
      File "twitter-stream-search.py", line 134, in
        os.execv('twitter-stream-search.py', [''])
    OSError: [Errno 13] Permission denied

    You have any idea what kind of problem is there?

    Reply

    • If you are running on Linux/MacOS, make sure that your script is executable, i.e. do

      $ chmod a+x twitter-stream-search.py

      When it complains about Exec format error, make sure that you have the following line at the beginning of the file:

      #!/usr/bin/env python

      Alternatively (or when you are on Windows), try changing the os.execv() line from

      os.execv('twitter-stream-search.py', [''])

      to

      os.execv(sys.executable, ['python', __file__])
      Reply

  8. Man thanks for your quickly respond.
    I am running on Ubuntu 14.04.

    I try both of your solutions but nothing change.

    itsoum@itsoum-Inspiron-3542:~/thesis$ sudo chmod a+x twitter-stream-search2.py
    itsoum@itsoum-Inspiron-3542:~/thesis$ sudo python twitter-stream-search2.py
    (13:08:45) @drewney2000 from None
               RT @UEFAEURO: Shaqiri's corner from the right is met by Schär, who
               climbs above Albania keeper Berisha to head #SUI in front. #EURO2016
               #AL…
    (13:08:46) @bintumariam from None
               RT @CuStudent: RT &amp; Predict the scoreline between #ENG  &amp;
               #RUS in today's #EURO2016 game. First 3 correct predictions get
               airtime courtesy…
    (13:08:46) @AbsoluteRSports from Seattle, WA
               RT @NBCSportsSoccer: 5 games coming your way today! #EURO2016,
               #CopaAmerica Today: England-Russia; #USMNT in pivotal clash
               https://t.co/e3v…
    Traceback (most recent call last):
      File "twitter-stream-search2.py", line 134, in <module>
        os.execv('twitter-stream-search.py', [''])
    OSError: [Errno 13] Permission denied

    You have any other idea?

    Thnx,
    I.T.

    Reply

    • From the traceback

      Traceback (most recent call last):
        File "twitter-stream-search2.py", line 134, in <module>
          os.execv('twitter-stream-search.py', [''])
      OSError: [Errno 13] Permission denied

      it seems that you did not change the name of the file in your script. Try changing the line

      os.execv('twitter-stream-search.py', [''])

      to

      os.execv('twitter-stream-search2.py', [''])

      and run the script via

      $ ./twitter-stream-search2.py
      Reply

  9. Excellent….
    Simply this was excellent.I went on finding for days to make a restart method for my GUI brain game.I found no way.But simply two lines from your dictums helped me out.

    Reply

  10. This is perfect!!! Thank you much…!!!
    I’ve put it in my excepthook function, adding the feature to control how many times should it be restarted before stopping it. Maybe this could be useful :D

    ## CODE TO RESTART python SCRIPT n TIMES then STOP IT     
    ## the last argument has to be 0 the 1st time it's launched
    ## Ex: python myscript.py <MY_ARGUMENTS> 0
     
    import sys
    import os
     
    def myexcept(t, v, tb):
       import traceback
       ## edit to change max times to restart
       max_times = 5
       ## grabs the exception details
       tbtext = ''.join(traceback.format_exception(t, v, tb))
       print " ERROR:"
       ## prints exception details
       print tbtext
       restarted = int(restarted)+1
       if restarted > max_times:
            ## stops script if restarted more than max_times
            print " STOPPED!!!"
            exit(0)
        else:
            print " RESTARTED "+str(restarted)+" times..."
            ## sets new argument list updating last argv
            a = ['python']+sys.argv[:-1]+[str(restarted)]
            ## restarts the script
            os.execv( sys.executable, a)
     
    ## set myexcept as global script exception method
    sys.excepthook = myexcept
     
    ## grabs times the scripts has been restarted (last argument)
    restarted = sys.argv[-1]
    print " RESTARTED "+restarted+" times :-)" )
    Reply

  12. Hi, thanks a lot for sharing this, it helped me a lot. At the same time, I have encountered some issues with it as well, wondering if you could help me. I created a button with a tkinter gui, when I press the button, it would run the code

    os.execv(sys.executable, ['python'] + sys.argv)

    . It worked perfectly for the first time, but when I press the same button on the restarted tkinter gui, the gui was closed (which is good), the new one did not show up. Within the console, it did not show anything, it was still recognized as the program is running i guess. Which made me press “control c” to interrupt it, then it says

    Fatal Python error: Py_Initialize: can’t initialize sys standard streams

    Do you have any ideas? Than you so much!

    Reply

    • Ha! I have managed to restart the program in another way, but I think its a way that everybody knows.
      I simply just use the following code.

      os.startfile('file_name')
      os.exit()

      I know it is like a “primary school” stuff, but it worked for me well, thanks anyways.

      Reply
  13. #!/usr/bin/env python
     
    import time
    import sys
    import os
    from datetime import datetime
     
     
     
    while True:
     
        now = datetime.now().time()
        print (now.strftime("%H:%M:%S"), end = "", flush=True),
        print("\r", end ='', flush= True),
        time.sleep(1)
     
        if now.hour == 13 and now.minute == 00 and now.second == 00:
            os.execv("/home/pi/Desktop/rasp_2016/dummy_code.py", sys.argv)
            print("rebooted")
            exit

    I want to restart a script at a particular time for that I have made a above code and I have executed “chmod a+x dummy_code.py” also but still getting an [Errorno 8] Exec format error. Please let me what could be the possible problem.

    Reply

Leave a Reply to Kushagra Rathi Cancel reply