EMBL importing issue
Open
EMBL importing issue
Hi,
I have an issue which seems to be similar or the same as issue #129.
I am trying to import the latest EMBL release using the methods outlined in the wolf tutorial but I get this error below. on this file Parsing this file STD_PLN_4.dat.gz. I have tired removing the file to see if the issue only this file is the issue but I also get the fault with STD_VRL_11.dat.gz.
I have tried updating obitools using pip and currently have version 3.0.1b22 which I hope is the latest version but it doesn't seem to resolve the issue for me at least.
The sequence that cannot be imported is very long so I just attached the start and the end which I think are the only bits which may have useful information.
DEBUG /private/var/folders/hx/z6wbn36d04l48hw501jt40lm0000gn/T/pip-install-gxjlmvbt/obitools3_e27c130ed86d4de0aeeea00f0794a369/src/obiavl.c:1669:obi_create_avl, obi_errno = 20, errno = 24 : Error creating an AVL tree file DEBUG /private/var/folders/hx/z6wbn36d04l48hw501jt40lm0000gn/T/pip-install-gxjlmvbt/obitools3_e27c130ed86d4de0aeeea00f0794a369/src/obiavl.c:1013:add_new_avl_in_group, obi_errno = 20, errno = 24 : Error creating a new AVL tree in a group
Could not import sequence: {b'ID': b'LR812263', b'NUC_SEQ': b'CAAATCTAGCAAGAATAGGCCAAAACTACAAGTTTTGAGGAGTTCCCCGTAACTGGACCCCGAGGTTCCCGAAATGTTTGGATCACAGCGGGACACTAAATCAGTGACTAATAACATACAAAATTGTCTGCAGTAGTCCTAAACTGCGAGTTTTGACGAGTTCCACGTAACCGGACCCCGAGGTTCCTGAAACATCCGGATCGTAGCGGGACCCAAAATCAATGAGTAATTGCATACAAAACTGGCAAGAACAGGCCAAAACTGCGAGTTTTAACGAGTTTTCGGTAACCGGACCCCGGGGGTCCCGAAACATTCGTATCGCAGGGAAACCAAAAACAGTGACTAATGGCCTAGAAAACTAGCCAGAACAGGCCAACACTGTGAGTTTTCACGAATTCCCCGTAACCCCACCTCGGGGTTCCCGAAACGTTCGGATCGCAGCGGGACCCAAAATGAGTGAGTAATAGCATATAAAACTAGCCAGAATAGGCCAAAACTGCGAGTTTGAAGAGTTCCCCGTAACCACACCCCGGGGTTCCCGAAATGTTCGGATCGTAGTAGGACCCCAAATCAGTCAGTAATAGCATACAAATCTAGCAAGAATAGGCCAAAACTACAAGTTTTGAGGAGTTCCCCGTAACTGGACCCCGAG
.........
GACACGCTCGTCGGACACCGGCAGGGCAGCTAGCTAGTCTGTGCGCGTCGCGACTCCCTTCACCAGCCGTCTTCTTCGTCAACGCTCGCAAGTTGTTCGACGGTTTGCCAAGGTACAAAATGGACTCCGTCGACGAGTTCTTTTTTCACAATTTCCTTTGCGACTCCGACGATTCGTCATCCGATGACGAGGAGGAGGTATTGGCTGCCGTGTTGGTCCATCACCTGCTCAATAGCTAGCGGCCGTTGTTCCGTGGCTCCATTCCGAGCCACCTTCCGGTGTTGA', b'DEFINITION': b'Hordeum vulgare subsp. vulgare genome assembly, chromosome: 3H', b'TAXID': 112509, b'organism': b'Hordeum vulgare subsp. vulgare'} Error raised: Problem setting a value in a column /!\ Check if '--input-na-string' option needs to be set zsh: segmentation fault obi import --embl-input EMBL Guate/embl_refs
I hope I have included the relevant info, and any help would be greatly appreciated!
Best,
Tom
I also have an issue with importing things into obitools more generally no matter the file type it will not import even the tutorial files whether they be the fastq or the nfs files. I don't know if the issues are related or if I am doing something fundamentally wrong.
2023-08-24 16:32:14,445 [import : INFO ] Opened file: wolf_tutorial/wolf_F.fastq 2023-08-24 16:32:14,470 [import : INFO ] Importing 45276 entries 72.4 % |####################################| ] remain : 00:00:0StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Users/thomashughes/obi3-env/bin/obi", line 62, in config[root_config_name]['module'].run(config) File "python/obitools3/commands/import.pyx", line 333, in obitools3.commands.import.run File "python/obitools3/parsers/fastq.pyx", line 106, in fastqWithQualityIterator RuntimeError: generator raised StopIteration
I hope that is helpful and not making things more complicated!
Tom
I also get the same error @tdhughes18. Trying to import demultiplexed files in a for loop, each one hits the
RuntimeError: generator raised StopIteration
. Interestingly, if I use--only
flag for a hard cutoff (less than the max number of entries), the seqs appear to import.Tim
Edited by Timothy DivollHi, the StopIteration issue was caused by a change in the behaviour of StopIteration exceptions in Python>=3.7, it should be fixed in the last version: https://git.metabarcoding.org/obitools/obitools3/-/wikis/Updating-and-release-notes#release-notes
I'm also looking into the 'too many files open' error that some people run into, but it's a tricky one.
Cheers Celine