python - PostgreSQL connection closes unexpectedly when doing a large insert -
i'm populating postgresql table ~11.000.000 rows have been selected before database. i'm using python , psycopg2. whole process takes estimated 1.5 hours complete. however, after ~30 minutes "connection closed unexpectedly" exception. source code looks this: incursor = indb.cursor() incursor.execute("select ...") indb.commit() # (1) close transaction outcursor = outdb.cursor() rows = 0 (col1, col2, col3) in incursor: # incursor contains ~11.000.000 rows outcursor.execute("insert ...", (col1, col2, col3)) # fails after ~30 minutes row += 1 if row % 100 == 0: # (2) write data every 100 rows outcursor.close() outdb.commit() outcursor = outdb.cursor() incursor.close() outcursor.close() outdb.commit() i inserted (1) , (2) after first tries failed, assuming open transaction has upper time limit of ~30 minutes or cursor has upper limit of pending inserts. seems none of assumptions true , error lies somewher...