Celine Mercier (3db93ee9) at 12 Jan 03:32
Fixed stdout output
Celine Mercier (6819d046) at 29 Nov 03:25
Testing RAM instead of mmap for blob alignment
Celine Mercier (fb6e27bb) at 29 Nov 03:22
Revert "Testing RAM instead of mmap for blob alignment"
Celine Mercier (6d94cdcc) at 29 Nov 03:20
Testing RAM instead of mmap for blob alignment
Yes, you can find examples in the wolf tutorial (https://git.metabarcoding.org/obitools/obitools3/-/wikis/Wolf-tutorial-with-the-OBITools3)
Hi, the StopIteration issue was caused by a change in the behaviour of StopIteration exceptions in Python>=3.7, it should be fixed in the last version: https://git.metabarcoding.org/obitools/obitools3/-/wikis/Updating-and-release-notes#release-notes
I'm also looking into the 'too many files open' error that some people run into, but it's a tricky one.
Cheers Celine
A group of us are trying to run the wolf tutorial and we all get an error trying to import the sequences into a DMS:
2023-09-08 15:03:44,255 [import : INFO ] obi import: imports an object (file(s), obiview, taxonomy...) into a DMS
2023-09-08 15:03:44,258 [import : INFO ] Opened file: wolf_tutorial/wolf_F.fastq
2023-09-08 15:03:44,300 [import : INFO ] Importing 45276 entries
72.4 % |####################################| ] remain : 00:00:0StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/tdivoll/.pyenv/versions/3.8.10/bin/obi", line 62, in <module>
config[root_config_name]['module'].run(config)
File "python/obitools3/commands/import.pyx", line 333, in obitools3.commands.import.run
File "python/obitools3/parsers/fastq.pyx", line 106, in fastqWithQualityIterator
RuntimeError: generator raised StopIteration```
I can get the seqs to import if I use the --only flag for one less than the total number of sequences:
2023-09-08 15:04:12,641 [import : INFO ] obi import: imports an object (file(s), obiview, taxonomy...) into a DMS
2023-09-08 15:04:12,643 [import : INFO ] Opened file: wolf_tutorial/wolf_F.fastq
2023-09-08 15:04:12,685 [import : INFO ] Importing 45275 entries
100.0 % |##################################################/] remain : 00:00:0
2023-09-08 15:04:13,115 [import : INFO ] Imported 45275 entries
2023-09-08 15:04:13,131 [import : INFO ] Done.
Could this be a bug? It seems like obi is looking for more sequences once it hits the end and then throws the error. We tested this locally on Mac M1 as well as on our HPC, with python versions 3.9.0 and 3.8.10. Interestingly, we get the same StopIteration error trying to import the wolf ngsfilter into the DMS, but the reference sequences import just fine.
Hi, this was caused by a change in the behaviour of StopIteration exceptions in Python>=3.7, it should be fixed in the last version: https://git.metabarcoding.org/obitools/obitools3/-/wikis/Updating-and-release-notes#release-notes
Cheers Celine
Celine Mercier (daf43e6c) at 21 Sep 05:49
obi import: fixed bug caused by new behaviour of StopIteration
Celine Mercier (8a1f8446) at 21 Sep 05:47
obi import: fixed bug caused by new behaviour of StopIteration
Oops! My apologies, obi convert is not in fact available in the OBI3, as it should not be needed. Why would you like to use it?
Hi Tiff,
If you reformat those sequence headers to have the scientific name in a tag as such:
>OK183856.1 SCIENTIFIC_NAME=Cercopithecus erythrotis; voucher T1768 16S large subunit ribosomal RNA gene, partial sequence; mitochondrial
Then you can use obi addtaxids
which adds the missing taxid information:
obi import /ref_dbs/nsdpy/NSDPY_results/2023-08-11_12-06-02/fasta/sequences.fasta iDNAtest/nsdpy_refs
wget https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz
obi import --taxdump taxdump.tar.gz iDNAtest/taxonomy/my_tax
obi addtaxids --taxonomy iDNAtest/taxonomy/my_tax iDNAtest/nsdpy_refs iDNAtest/nsdpy_refs_with_taxids
obi ecopcr -e 3 -l 50 -L 160 -F CGGTTGGGGTGACCTCGGA -R GCTGTTATCCCTAGGGTAACT --taxonomy iDNAtest/taxonomy/my_tax iDNAtest/nsdpy_refs_with_taxids iDNAtest/16SMam_refs
Cheers
Celine
Hi,
When running obi build_ref_db --taxonomy Coll01_r140_stripped/taxonomy/my_tax Coll01_r140_stripped/Coll01_r140_refs_clean Coll01_r140_stripped/Coll01_r140_db_99
I get:
2023-08-21 10:57:00,647 [build_ref_db : INFO ] obi build_ref_db
Aligning sequences...
Building kmer tables... : Done
Error opening the taxid column when building a reference database
Traceback (most recent call last):
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/bin/obi", line 4, in <module>
__import__('pkg_resources').run_script('OBITools3==3.0.1b24', 'obi')
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/lib/python3.10/site-packages/pkg_resources/__init__.py", line 722, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1561, in run_script
exec(code, namespace, namespace)
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/lib/python3.10/site-packages/OBITools3-3.0.1b24-py3.10-linux-x86_64.egg/EGG-INFO/scripts/obi", line 62, in <module>
config[root_config_name]['module'].run(config)
File "python/obitools3/commands/build_ref_db.pyx", line 90, in obitools3.commands.build_ref_db.run
Exception: Error building a reference database
I used the standard ncbi taxdump, and the fasta I used as import for the reference db sequences looks like this:
>AM263556
taatttttattaattctttttaaggattatttacctaacggtagttattttatatggagt
tttagatgctccatggttgccccaacaaaatagtatctcaatctatatctacaataagat
tgagagtgctataaaac
>AM263557
taatttttattaattctttttaaggattatttacctaacggtagttattttatatggagt
tttagatgctccatggttgccccaacaaaatagtacctcaatctatatctacaataagat
tgagagtgctataaaac
>AM263558
taatttttattaattctttttaaggattatttacctaacggtagttattttatatggagt
tttagatgctccatggttgccccaacaaaatagtatctcaatctatatctacaataagat
tgagagtgctataaaac
>AM263559
taatttttattaattctttttaaggattatttacctaacggtagttattttatatggagt
tttagatgctccatggttgccccaacaaaatagtacctcaatctatatctacaataagat
tgagagtgctataaaac
Any idea what I am doing wrong?
Kind regards, Sam
Solved
Hi,
I'm trying to assign taxonomy to my sequences like so:
obi ecotag -m 0.60 --taxonomy Coll01_r140_stripped/taxonomy/my_tax -R Coll01_r140_stripped/Coll01_r140_refs_clean Coll01_r140_stripped/otu85_sequences Coll01_r140_stripped/assigned_sequences
But I get the following error message:
Traceback (most recent call last):
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/bin/obi", line 4, in <module>
__import__('pkg_resources').run_script('OBITools3==3.0.1b24', 'obi')
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/lib/python3.10/site-packages/pkg_resources/__init__.py", line 722, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1561, in run_script
exec(code, namespace, namespace)
File "/kyukon/home/gent/462/vsc46214/obitools3/obi3-env/lib/python3.10/site-packages/OBITools3-3.0.1b24-py3.10-linux-x86_64.egg/EGG-INFO/scripts/obi", line 62, in <module>
config[root_config_name]['module'].run(config)
File "python/obitools3/commands/ecotag.pyx", line 80, in obitools3.commands.ecotag.run
File "python/obitools3/dms/view/view.pyx", line 704, in obitools3.dms.view.view.View_comments.__getitem__
KeyError: b'ref_db_threshold'
In trying to figure out what might be causing the ref_db_treshold
error, the only thing that comes to mind is that I didn't run obi build_ref_db
, because that didn't work and gave an error. Is obi build_ref_db
needed to be able to run obi ecotag
?
For reference, I used the following commands to build the db:
obi import --fasta-input ./otus85_stripped.fasta Coll01_r140_stripped/otu85_sequences
obi import --taxdump taxdump.tar.gz Coll01_r140_stripped/taxonomy/my_tax
obi import --fasta-input ./Coll01_r140_cleaned.fasta Coll01_r140_stripped/Coll01_r140_refs
obi grep --require-rank=species --require-rank=genus --require-rank=family --taxonomy Coll01_r140_stripped/taxonomy/my_tax Coll01_r140_stripped/Coll01_r140_refs Coll01_r140_stripped/Coll01_r140_refs_clean
obi ecotag -m 0.60 --taxonomy Coll01_r140_stripped/taxonomy/my_tax -R Coll01_r140_stripped/Coll01_r140_refs_clean Coll01_r140_stripped/otu85_sequences Coll01_r140_stripped/assigned_sequences
Kind regards, Sam
Solved (ref db not built)
Hi Gustavo,
Commands are in 2 words with the OBITools3, try:
obi convert -h
Cheers Celine