Restarting a Python Script Within Itself

By Petr Zemek in Software Engineering March 23, 2014 29 Comments

Sometimes, you may wish to check within a script when a configuration file or the script itself changes, and if so, then automatically restart the script. In this post, you will see a way of doing this in Python.

Consider the following scenario. You have a Python script that runs as a daemon and regularly performs the prescribed tasks. Examples may be a web server, a logging service, and a system monitor. When the script starts, it reads its configuration from a file, and then enters an infinite loop. In this loop, it waits for inputs from the environment and acts upon them. For example, a web server may react to a request for a page, which results into sending a response to the user.

From time to time, it may be necessary to restart the script. For example, if you fix a bug in it or change its configuration. One way of doing so is to kill the script and run it again. However, this requires manual intervention, which you may forget to do. When you fix a vulnerability in the script, you want to be sure that you do not forget to restart the script. Otherwise, someone may exploit the vulnerability if you did not restart the script. It would be nice if there existed a way of restarting the script within itself after it detected that its sources or a configuration file changed. In the rest of this post, we will show such a way.

For the purpose of the present post, let us assume that the script has the following structure:

# Parse the arguments and configuration files.

while True:
    # Wait for inputs and act on them.
    # ...

That is, it processes the arguments and loads the configuration from the configuration files. After that, the script waits for inputs and processes them in an infinite loop.

Next, we describe how to watch files for changes. After that, we show how to restart the script.

Checking Watched Files For Changes

First, we define the paths to the files whose change we want to watch:

WATCHED_FILES = [GLOBAL_CONFIG_FILE_PATH, LOCAL_CONFIG_FILE_PATH, __file__]

We watch the global configuration file, the local configuration file, and the script itself, whose path can be obtained from the special global variable __file__. When the script starts, we get and store the time of the last modification of these files by using os.path.getmtime():

from os.path import getmtime

WATCHED_FILES_MTIMES = [(f, getmtime(f)) for f in WATCHED_FILES]

Then, we add a check if any of these files have changed into the main loop:

while True:
    for f, mtime in WATCHED_FILES_MTIMES:
        if getmtime(f) != mtime:
            # Restart the script.

    # Wait for inputs and act on them.
    # ...

If either of the files that we watch has changed, we restart the script. The restarting is described next.

Restarting the Script

We restart the script by utilizing one of the exec*() functions from the os module. The exact version and arguments depend on how you run the script. For example, on Linux or Mac OS, you can make the file executable by putting the following line to the top of the file

#!/usr/bin/env python

and executing

$ chmod a+x daemon.py

Then, you can run the script via

$ ./daemon.py

In such a situation, to restart the script, use the following code:

os.execv(__file__, sys.argv)

Otherwise, when you run the script via

$ python daemon.py

use this code:

os.execv(sys.executable, ['python'] + sys.argv)

Either way, do not forget to import the sys module:

import sys

To explain, the arguments of os.execv() are the program to replace the current process with and arguments to this program. The __file__ variable holds a path to the script, sys.argv are arguments that were passed to the script, and sys.executable is a path to the Python executable that was used to run the script.

The os.execv() function does not return. Instead, it starts executing the current script from its beginning, which is what we want.

Concluding Remarks

If you use the solution above, please bear in mind that the exec*() functions cause the current process to be replaced immediately, without flushing opened file objects. Therefore, if you have any opened files at the time of restarting the script, you should flush them using f.flush() or os.fsync(fd) before calling an exec*() function.

Of course, the presented solution is only one of the possible ways of restarting a Python script. Depending on the actual situation, other approaches, like killing the script externally and starting it afterwards, may be more suitable for you. Moreover, there exist other methods of checking whether a watched file has changed and acting upon such a change. If you know of another way of restarting a Python program within itself, please share it by posting a comment.

Complete Source Code

The complete source code for this post is available on GitHub.

29 Comments

Justin

March 28, 2015 at 16:53
Petr,

This looks to be exactly what I am looking for.
I am very new to Python though so I am struggling figuring out how exactly to implement this.
I have a cron job that runs (lets call it program A) every night at midnight and goes to a webservice and checks to see if i have updated my code to my program (lets call it program B). If I have, it downloads the code and overwrites my program B code. Until know I couldn’t get program A to restart program B. So I set up a cron job that just reboots the PI every night. I don’t like this and would really like Program B to pick up the change and restart itself only if there has been a change. I am struggling on where to put this while loop in my code.

lets say i have a stop watch program that essentially shows a clock and a stop watch at the same time. Its always waiting on external inputs from GPIO to start and stop the stop watch and record their times. I tried putting your loop at the top and the bottom but my code either goes into an infinite loop or doesn’t display the main program. Any advice you could give would be appreciated.
```
  
def main():
    root = Tk()
    FS = FullScreenApp(root)
    cf = ConfigFile(root)
    frame = Frame(root)
    app = App(root)
    sw = StopWatch(root)
    root.mainloop()
```
Reply
- Petr Zemek
  
  March 29, 2015 at 14:07
  Hi Justin!
  
  You have to put the checking and restarting into a place that is periodically executed. For example, in a Tk application, this can be done by using root.after() as follows:
```
#!/usr/bin/env python

import os
import sys
# If you use Python 2, replace tkinter with Tkinter.
from tkinter import Tk

# Number of milliseconds after which a check whether
# the file has changed is performed.
CHECK_INTERVAL = 2000

# Last time the file was modified.
last_mtime = os.path.getmtime(__file__)

# The main window.
root = Tk()

def restart_if_changed():
    # Check if the file has changed.
    # If so, restart the application.
    if os.path.getmtime(__file__) > last_mtime:
        print('file has changed => restarting')

        # Restart the application (os.execv() does not return).
        os.execv(__file__, sys.argv)

    # Reschedule the checking.
    root.after(CHECK_INTERVAL, restart_if_changed)

# Schedule the checking.
root.after(CHECK_INTERVAL, restart_if_changed)

# Enter the main processing loop.
root.mainloop()
```
  Just save the code into a file, make the file executable, and run it. Then, whenever you modify the file (its mtime attribute is changed), the application is automatically restarted. You can try it by yourself: run the application, modify the file in a text editor, and after at most two seconds, the application should be restarted.
  Reply
  - Paul Frischknecht
    
    May 3, 2016 at 16:10
    
    But what if I accidentally introduce an error in the new version of this .py file? I would want the old version to keep on running (and output the error), and *try* restarting again when the file is next updated.
    
    Your solution would have to be manually restarted after the problem is fixed.
    
    Reply
Bill

April 24, 2015 at 19:06
Hello Petr

I’m running a script on a beaglebone black with debian
Here is the part that gets the error.
```
for f, mtime in watched_files_mtimes:
  if getmtime(f) > mtime:
  writeProgLog("File change")
  print "Restarting program! -- file changed"
  os.execv(__file__, sys.argv)
threading.Timer(20, every20Seconds).start()
```
this is the error
```
    os.execv(__file__, sys.argv)
OSError: [Errno 13] Permission denied
```
PS. I guess I need to figure out how to post a code block
thanks
bill
Reply
- Petr Zemek
  
  April 25, 2015 at 10:22
  Hi Bill! Make sure that your script is executable, i.e. do
```
chmod a+x script.py
```
  Then, the code you posted should work (I tested it).
  Reply
  - erm3nda
    
    August 22, 2015 at 08:10
    Hi. Once I make the python file executable I got a new error:
    
    Traceback (most recent call last): File "file.py", line 30, in os.execv(__file__, sys.argv) OSError: [Errno 8] Exec format error
    
    I’ve checked and tried some shebangs but nothing works.
    
    I don’t know what os.execv(__file__, sys.argv) exacly does, but seems a popen command at all.
    This is what i did to relaunch the script:
    
    def relaunch(): args = "" # must declare first for i in sys.argv: args = args + " " + str(i) # force casting to strings with str() to concatenate print "python" + args # print what would we run os.system("python" + args) # do it!
    
    Python script calling relaunch will launch itself again. I’m sure this solution do not cover memory handling or other aspects, but I need the script reloading at any cost :D
    Reply
    - Petr Zemek
      
      August 26, 2015 at 08:59
      
      Hi erm3nda. To fix the Exec format error, put the following line to the top of the file:
      
      #!/usr/bin/env python
      
      This line informs the system to run the script in Python. When this line is not present, the system does not know how to execute the script when it is relaunched.
      
      Reply
    - Michael
      
      October 18, 2017 at 04:46
      
      Also, what you are doing is blocking your script with an error and launching a new instance within that.
      
      Reply
- Edward
  
  August 2, 2015 at 08:00
  An easy way to restart the Python Shell programmatically is as follows
```
import _winapi
x = _winapi.GetCurrentProcess()
def RESTART() : _winapi.ExitProcess(x)
RESTART()
```
  That was one way I found.
  Reply
erm3nda

September 18, 2015 at 17:24

@Petr Zemek about “#!/usr/bin/env python”

I saw that I tried some shebangs and still doesn’t work.
This time im working on a Windows machine and shebang does exactly nothing on it.

I’ve checked .py assoc with pythonw.exe instead python.exe and still doesn’t work.
I’m really stucked with that because that would be dead simple and is not.

Another way would be to create a simple Launcher just for that.

Reply
Ronald

November 5, 2015 at 10:38
Nice code, just what I was looking for. Thanks!
What if the new script does eventually not work as expected and you revert back to the backed-up original? In that case
```
if getmtime(f) > mtime:
```
prevents the restart because the backup file has a mtime that is < than the mtime of the running script.
I would say: use
```
if getmtime(f) <> mtime:
```
Reply
- Petr Zemek
  
  November 8, 2015 at 09:10
  
  Yes, you are right, != is better. I have updated the code. Thanks!
  
  Reply
Oliver

December 30, 2015 at 06:43
Hi Petr

1st of all, let me thank you very much for your explanation. I inserted your code fragment into a timer. There are two routines: 1 routine compares the NOW-time to the saved times in a list and switches the respective GPIO, when NOW- and SAVE-time are the same. This routine runs Daemon-like in the background and is started by a cronjob @reboot. The other routine let’s me change the saved times (switch times) and the names of the channels. Now, when I’m changing times or names, I save the new list with ‘pickle’ to a file. When this file is changed, the switch-routine restarts itself and reloads the pickle file to adopt the new settings. I know, it’s surely not the fine English style of coding, but it works flawlessly, that’s the essential.

2 things to remark: First of all, your code runs flawlessly unter Python 3 as well. Just use
```
#!/usr/bin/python3
```
as interpreter description (I assume it is installed…). Secondly, you can check every single file you want. Just use
```
WATCHED_FILES = ['/home/user/.../nameofthescript.py']
```
Just for the beginners who don’t know – like me. (Trial and error – method…) :)
Reply
- Visor
  
  May 26, 2016 at 19:13
  
  I also got the same problem.
  It seems that __file__ refers to the compiled bytecode file with .pyc instead of the original .py.
  Does this break the os.exec statement when trying to exec the bytecode directly?
  
  Reply

Here’s my solution. It can relaunch the file even after you introduced errors.

# usage:
#   RelaunchWatchdog.py <subfile>.py

# This program takes a .py file which it immediately executes in a subprocess.
# It kills the subprocess and relaunches it whenever the file changes.
#
# This program is independent of the python file it launches: 
# If you accidentally introduce an error into the .py file, just fix that error and save it again, and it will be relaunched.
# Note that the subprocess shared the console with this process, so you will see its output.

import sys,time,subprocess,logging, os, sys, watchdog, watchdog.events, watchdog.observers,psutil

subfile = sys.argv[1]

def launch():
  global process
  process = subprocess.Popen(subfile, shell=True)

def samefile(a,b):
  return os.stat(a) == os.stat(b)

def kill_or_do_nothing():
  global process
  try: # Process might already have terminated
    process = psutil.Process(process.pid)
    for proc in process.get_children(recursive=True):
      proc.kill()
    process.kill()
  except:
    pass
    
class Modified(watchdog.events.FileSystemEventHandler):
    def on_modified(self, event):
        if not samefile(event.src_path, subfile):
            return
        
        print subfile, 'modified, will restart'
        kill_or_do_nothing()
        launch()

observer = watchdog.observers.Observer()
observer.schedule(Modified(), '.')
observer.start()

launch()

# Idle forever, listening on Modified.on_modified
while True:
  time.sleep(1)

Ilias Tsoumas

June 10, 2016 at 15:05
man i will take back this
```
Traceback (most recent call last):
  File "twitter-stream-search.py", line 134, in 
    os.execv('twitter-stream-search.py', [''])
OSError: [Errno 13] Permission denied
```
You have any idea what kind of problem is there?
Reply
- Petr Zemek
  
  June 10, 2016 at 15:37
  If you are running on Linux/MacOS, make sure that your script is executable, i.e. do
```
$ chmod a+x twitter-stream-search.py
```
  When it complains about Exec format error, make sure that you have the following line at the beginning of the file:
```
#!/usr/bin/env python
```
  Alternatively (or when you are on Windows), try changing the os.execv() line from
```
os.execv('twitter-stream-search.py', [''])
```
  to
```
os.execv(sys.executable, ['python', __file__])
```
  Reply

Man thanks for your quickly respond.
I am running on Ubuntu 14.04.

I try both of your solutions but nothing change.

itsoum@itsoum-Inspiron-3542:~/thesis$ sudo chmod a+x twitter-stream-search2.py
itsoum@itsoum-Inspiron-3542:~/thesis$ sudo python twitter-stream-search2.py
(13:08:45) @drewney2000 from None
           RT @UEFAEURO: Shaqiri's corner from the right is met by Schär, who
           climbs above Albania keeper Berisha to head #SUI in front. #EURO2016
           #AL…
(13:08:46) @bintumariam from None
           RT @CuStudent: RT &amp; Predict the scoreline between #ENG  &amp;
           #RUS in today's #EURO2016 game. First 3 correct predictions get
           airtime courtesy…
(13:08:46) @AbsoluteRSports from Seattle, WA
           RT @NBCSportsSoccer: 5 games coming your way today! #EURO2016,
           #CopaAmerica Today: England-Russia; #USMNT in pivotal clash
           https://t.co/e3v…
Traceback (most recent call last):
  File "twitter-stream-search2.py", line 134, in <module>
    os.execv('twitter-stream-search.py', [''])
OSError: [Errno 13] Permission denied

You have any other idea?

Thnx,
I.T.

Petr Zemek

June 11, 2016 at 18:14
From the traceback
```
Traceback (most recent call last):
  File "twitter-stream-search2.py", line 134, in <module>
    os.execv('twitter-stream-search.py', [''])
OSError: [Errno 13] Permission denied
```
it seems that you did not change the name of the file in your script. Try changing the line
```
os.execv('twitter-stream-search.py', [''])
```
to
```
os.execv('twitter-stream-search2.py', [''])
```
and run the script via
```
$ ./twitter-stream-search2.py
```
Reply
- Ilias Tsoumas
  
  June 11, 2016 at 23:49
  thnx man… i see it and i fixed it. Now, the script run ok… i don’t run script via
```
$ ./twitter-stream-search2.py
```
  but with the previous method
```
python twitter-stream-search2.py
```
  Reply
  - Petr Zemek
    
    June 12, 2016 at 07:18
    
    Great! I’m glad I could help.
    
    Reply

Ayan Mukherjee

July 28, 2016 at 14:48

Excellent….
Simply this was excellent.I went on finding for days to make a restart method for my GUI brain game.I found no way.But simply two lines from your dictums helped me out.

Reply

This is perfect!!! Thank you much…!!!
I’ve put it in my excepthook function, adding the feature to control how many times should it be restarted before stopping it. Maybe this could be useful :D

## CODE TO RESTART python SCRIPT n TIMES then STOP IT      
## the last argument has to be 0 the 1st time it's launched
## Ex: python myscript.py <MY_ARGUMENTS> 0

import sys
import os

def myexcept(t, v, tb):
   import traceback
   ## edit to change max times to restart
   max_times = 5
   ## grabs the exception details
   tbtext = ''.join(traceback.format_exception(t, v, tb))
   print " ERROR:"
   ## prints exception details
   print tbtext
   restarted = int(restarted)+1
   if restarted > max_times:
        ## stops script if restarted more than max_times
        print " STOPPED!!!"
        exit(0)
    else:
        print " RESTARTED "+str(restarted)+" times..."
        ## sets new argument list updating last argv
        a = ['python']+sys.argv[:-1]+[str(restarted)]
        ## restarts the script
        os.execv( sys.executable, a)

## set myexcept as global script exception method
sys.excepthook = myexcept

## grabs times the scripts has been restarted (last argument)
restarted = sys.argv[-1]
print " RESTARTED "+restarted+" times :-)" )

anndy

March 28, 2017 at 16:24

it’s a big help to me , thanks a lot

Reply
FatalErrorsBad4Health

June 11, 2018 at 00:16
Hi, thanks a lot for sharing this, it helped me a lot. At the same time, I have encountered some issues with it as well, wondering if you could help me. I created a button with a tkinter gui, when I press the button, it would run the code
```
os.execv(sys.executable, ['python'] + sys.argv)
```
. It worked perfectly for the first time, but when I press the same button on the restarted tkinter gui, the gui was closed (which is good), the new one did not show up. Within the console, it did not show anything, it was still recognized as the program is running i guess. Which made me press “control c” to interrupt it, then it says

Fatal Python error: Py_Initialize: can’t initialize sys standard streams

Do you have any ideas? Than you so much!
Reply
- FatalErrorsBad4Health
  
  June 11, 2018 at 00:36
  Ha! I have managed to restart the program in another way, but I think its a way that everybody knows.
  I simply just use the following code.
```
os.startfile('file_name')
```
```
os.exit()
```
  I know it is like a “primary school” stuff, but it worked for me well, thanks anyways.
  Reply
Kushagra Rathi

July 27, 2020 at 13:13
```
#!/usr/bin/env python

import time
import sys
import os
from datetime import datetime



while True:

    now = datetime.now().time()
    print (now.strftime("%H:%M:%S"), end = "", flush=True),
    print("\r", end ='', flush= True),
    time.sleep(1)

    if now.hour == 13 and now.minute == 00 and now.second == 00:
        os.execv("/home/pi/Desktop/rasp_2016/dummy_code.py", sys.argv) 
        print("rebooted")
        exit
```
I want to restart a script at a particular time for that I have made a above code and I have executed “chmod a+x dummy_code.py” also but still getting an [Errorno 8] Exec format error. Please let me what could be the possible problem.
Reply
- Petr Zemek
  
  July 27, 2020 at 19:44
  Hi. I suggest ensuring that /home/pi/Desktop/rasp_2016/dummy_code.py
  - begins with #!/usr/bin/env python, and
  - is executable.
  Reply
undefined

December 10, 2020 at 20:18

thank you so much sir !

Reply

Checking Watched Files For Changes

Restarting the Script

Concluding Remarks

Complete Source Code

29 Comments

Leave a Comment. Cancel reply