BrokenPipeError during oa index
Testing with version 0.1.0b3 compiled from source from git under 64bit Linux.
Example run with broken pipe error/warning (I have seen this multiple times with this data set):
[example_oa]$ time oa index --estimate-length=0.9 example_index example_R1.fastq example_R2.fastq > example_oa_index.log
2015-08-24 12:19:54,661 [INFO ] Computing read length statistics for the forward file
2015-08-24 12:36:52,244 [INFO ] Computing read length statistics for the reverse file
2015-08-24 12:53:44,513 [INFO ] Read size considered for a quantile of 0.90 : 73
2015-08-24 12:53:44,514 [INFO ] orgasmi -o example_index -l 73 /tmp/tmpkj35ci6j/forward-o2cuiuqh /tmp/tmpkj35ci6j/reverse-kixmu61t
2015-08-24 12:53:44,514 [INFO ] Starting indexing...
2015-08-24 12:53:45,226 [INFO ] Forward tmp file : /tmp/tmpkj35ci6j/forward-o2cuiuqh
2015-08-24 12:53:45,226 [INFO ] Reverse tmp file : /tmp/tmpkj35ci6j/reverse-kixmu61t
Reading sequence reads...
2015-08-24 12:54:08,140 [INFO ] 1000000 sequence pairs written
2015-08-24 12:54:29,335 [INFO ] 2000000 sequence pairs written
2015-08-24 12:54:49,971 [INFO ] 3000000 sequence pairs written
2015-08-24 12:55:11,959 [INFO ] 4000000 sequence pairs written
2015-08-24 12:55:35,231 [INFO ] 5000000 sequence pairs written
2015-08-24 12:55:59,460 [INFO ] 6000000 sequence pairs written
2015-08-24 12:56:20,811 [INFO ] 7000000 sequence pairs written
2015-08-24 12:56:45,078 [INFO ] 8000000 sequence pairs written
2015-08-24 12:57:05,474 [INFO ] 9000000 sequence pairs written
2015-08-24 12:57:24,555 [INFO ] 10000000 sequence pairs written
2015-08-24 12:57:46,636 [INFO ] 11000000 sequence pairs written
2015-08-24 12:58:08,629 [INFO ] 12000000 sequence pairs written
2015-08-24 12:58:30,383 [INFO ] 13000000 sequence pairs written
2015-08-24 12:58:52,942 [INFO ] 14000000 sequence pairs written
2015-08-24 12:59:12,730 [INFO ] 15000000 sequence pairs written
2015-08-24 12:59:34,018 [INFO ] 16000000 sequence pairs written
2015-08-24 12:59:54,031 [INFO ] 17000000 sequence pairs written
2015-08-24 13:00:13,337 [INFO ] 18000000 sequence pairs written
2015-08-24 13:00:35,218 [INFO ] 19000000 sequence pairs written
2015-08-24 13:00:55,414 [INFO ] 20000000 sequence pairs written
2015-08-24 13:01:15,856 [INFO ] 21000000 sequence pairs written
2015-08-24 13:01:37,387 [INFO ] 22000000 sequence pairs written
2015-08-24 13:01:57,091 [INFO ] 23000000 sequence pairs written
2015-08-24 13:02:19,429 [INFO ] 24000000 sequence pairs written
2015-08-24 13:02:43,850 [INFO ] 25000000 sequence pairs written
2015-08-24 13:03:07,314 [INFO ] 26000000 sequence pairs written
2015-08-24 13:03:29,556 [INFO ] 27000000 sequence pairs written
2015-08-24 13:03:50,060 [INFO ] 28000000 sequence pairs written
2015-08-24 13:04:09,522 [INFO ] 29000000 sequence pairs written
2015-08-24 13:04:30,661 [INFO ] 30000000 sequence pairs written
2015-08-24 13:04:49,922 [INFO ] 31000000 sequence pairs written
2015-08-24 13:05:09,162 [INFO ] 32000000 sequence pairs written
2015-08-24 13:05:31,030 [INFO ] 33000000 sequence pairs written
2015-08-24 13:05:53,046 [INFO ] 34000000 sequence pairs written
2015-08-24 13:06:13,792 [INFO ] 35000000 sequence pairs written
2015-08-24 13:06:33,918 [INFO ] 36000000 sequence pairs written
2015-08-24 13:06:55,409 [INFO ] 37000000 sequence pairs written
2015-08-24 13:07:18,337 [INFO ] 38000000 sequence pairs written
71428570 sequences read
Sorting reads...
Traceback (most recent call last):
File "/home/pc40583/lib/python3.4/site-packages/orgasm/command/index.py", line 401, in writeToFifo
print(formatFastq(p[1]),file=r)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pc40583/bin/oa", line 206, in <module>
config['orgasm']['module'].run(config)
File "/home/pc40583/lib/python3.4/site-packages/orgasm/command/index.py", line 541, in run
writeToFifo(pairs,forward,reverse,logger)
File "/home/pc40583/lib/python3.4/site-packages/orgasm/command/index.py", line 401, in writeToFifo
print(formatFastq(p[1]),file=r)
BrokenPipeError: [Errno 32] Broken pipe
Deleting tmp file : /tmp/tmpkj35ci6j/reverse-kixmu61t
Deleting tmp file : /tmp/tmpkj35ci6j/forward-o2cuiuqh
Deleting tmp directory : /tmp/tmpkj35ci6j
4088423 sequences sorted
real 47m52.235s
user 46m22.085s
sys 1m12.646s
71428570 sequences sorted example_oa]$ 4517256 sequences sorted
Writing sorted sequence reads...
71428570 sequences read
Writing sequence pairing data...
Done.
Reading indexed sequence reads...
71428570 sequences read
Sorting reads...
71428570 sequences sorted
Writing sequence suffix index...
Done.
Writing global data...
Done.
Possibly related, the stderr are not in the expected order (e.g. the time output).
The stdout captured to file was:
$ more example_oa_index.log
Counter({75: 29535694, 74: 14115528, 73: 3323060, 72: 1187185, 71: 275933, 70: 59113, 69: 23354, 68: 11722, 65: 9859, 67: 9370, 66: 8438, 64: 8273, 63: 8223
, 61: 7773, 62: 7553, 60: 5281, 59: 4865, 58: 3974, 57: 3064, 56: 2867, 55: 2762, 53: 2502, 52: 2493, 54: 2443, 51: 1926, 50: 1843, 47: 1794, 49: 1469, 42:
1189, 48: 1143, 45: 1127, 37: 1036, 44: 1022, 0: 1014, 29: 989, 27: 969, 46: 965, 41: 943, 22: 930, 43: 802, 21: 781, 30: 715, 28: 693, 31: 663, 40: 652, 26
: 640, 33: 611, 25: 610, 38: 567, 24: 553, 19: 513, 39: 486, 20: 408, 35: 385, 36: 361, 17: 347, 23: 328, 32: 295, 34: 283, 13: 174, 15: 166, 16: 116, 18: 1
06, 14: 45, 12: 37, 10: 35, 11: 10, 9: 1})
Counter({75: 31216287, 74: 12952469, 73: 2975735, 72: 1000256, 71: 244966, 70: 110509, 69: 24472, 68: 11879, 65: 9550, 67: 9342, 66: 8145, 64: 8089, 63: 735
1, 62: 6715, 61: 5835, 60: 5082, 59: 4768, 58: 3809, 0: 3198, 57: 3021, 55: 2700, 56: 2612, 52: 2253, 54: 2137, 53: 2085, 51: 2076, 50: 1740, 48: 1636, 49:
1512, 47: 1281, 45: 1041, 35: 877, 41: 851, 46: 797, 44: 719, 27: 678, 42: 677, 39: 665, 17: 633, 8: 627, 43: 611, 38: 586, 40: 575, 37: 504, 30: 500, 23: 4
98, 16: 459, 36: 447, 15: 446, 9: 423, 12: 411, 5: 410, 6: 399, 33: 395, 19: 374, 20: 370, 28: 360, 34: 356, 24: 320, 21: 301, 14: 286, 25: 285, 29: 269, 10
: 265, 32: 254, 26: 250, 13: 242, 11: 236, 18: 235, 7: 224, 31: 220, 22: 219, 4: 203, 3: 48, 2: 15})
maximum reads : 71428571
The log file is:
$ more example_index.log
2015-08-24 12:19:54,661 [INFO ] Computing read length statistics for the forward file
2015-08-24 12:36:52,244 [INFO ] Computing read length statistics for the reverse file
2015-08-24 12:53:44,513 [INFO ] Read size considered for a quantile of 0.90 : 73
2015-08-24 12:53:44,514 [INFO ] orgasmi -o SMFRLBV2_index -l 73 /tmp/tmpkj35ci6j/forward-o2cuiuqh /tmp/tmpkj35ci6j/reverse-kixmu61t
2015-08-24 12:53:44,514 [INFO ] Starting indexing...
2015-08-24 12:53:45,226 [INFO ] Forward tmp file : /tmp/tmpkj35ci6j/forward-o2cuiuqh
2015-08-24 12:53:45,226 [INFO ] Reverse tmp file : /tmp/tmpkj35ci6j/reverse-kixmu61t
2015-08-24 12:54:08,140 [INFO ] 1000000 sequence pairs written
2015-08-24 12:54:29,335 [INFO ] 2000000 sequence pairs written
2015-08-24 12:54:49,971 [INFO ] 3000000 sequence pairs written
2015-08-24 12:55:11,959 [INFO ] 4000000 sequence pairs written
2015-08-24 12:55:35,231 [INFO ] 5000000 sequence pairs written
2015-08-24 12:55:59,460 [INFO ] 6000000 sequence pairs written
2015-08-24 12:56:20,811 [INFO ] 7000000 sequence pairs written
2015-08-24 12:56:45,078 [INFO ] 8000000 sequence pairs written
2015-08-24 12:57:05,474 [INFO ] 9000000 sequence pairs written
2015-08-24 12:57:24,555 [INFO ] 10000000 sequence pairs written
2015-08-24 12:57:46,636 [INFO ] 11000000 sequence pairs written
2015-08-24 12:58:08,629 [INFO ] 12000000 sequence pairs written
2015-08-24 12:58:30,383 [INFO ] 13000000 sequence pairs written
2015-08-24 12:58:52,942 [INFO ] 14000000 sequence pairs written
2015-08-24 12:59:12,730 [INFO ] 15000000 sequence pairs written
2015-08-24 12:59:34,018 [INFO ] 16000000 sequence pairs written
2015-08-24 12:59:54,031 [INFO ] 17000000 sequence pairs written
2015-08-24 13:00:13,337 [INFO ] 18000000 sequence pairs written
2015-08-24 13:00:35,218 [INFO ] 19000000 sequence pairs written
2015-08-24 13:00:55,414 [INFO ] 20000000 sequence pairs written
2015-08-24 13:01:15,856 [INFO ] 21000000 sequence pairs written
2015-08-24 13:01:37,387 [INFO ] 22000000 sequence pairs written
2015-08-24 13:01:57,091 [INFO ] 23000000 sequence pairs written
2015-08-24 13:02:19,429 [INFO ] 24000000 sequence pairs written
2015-08-24 13:02:43,850 [INFO ] 25000000 sequence pairs written
2015-08-24 13:03:07,314 [INFO ] 26000000 sequence pairs written
2015-08-24 13:03:29,556 [INFO ] 27000000 sequence pairs written
2015-08-24 13:03:50,060 [INFO ] 28000000 sequence pairs written
2015-08-24 13:04:09,522 [INFO ] 29000000 sequence pairs written
2015-08-24 13:04:30,661 [INFO ] 30000000 sequence pairs written
2015-08-24 13:04:49,922 [INFO ] 31000000 sequence pairs written
2015-08-24 13:05:09,162 [INFO ] 32000000 sequence pairs written
2015-08-24 13:05:31,030 [INFO ] 33000000 sequence pairs written
2015-08-24 13:05:53,046 [INFO ] 34000000 sequence pairs written
2015-08-24 13:06:13,792 [INFO ] 35000000 sequence pairs written
2015-08-24 13:06:33,918 [INFO ] 36000000 sequence pairs written
2015-08-24 13:06:55,409 [INFO ] 37000000 sequence pairs written
2015-08-24 13:07:18,337 [INFO ] 38000000 sequence pairs written