OpenMS: FileFilter

Extracts portions of the data from an mzML, featureXML or consensusXML file.

pot. predecessor tools

$\longrightarrow$ FileFilter $\longrightarrow$

pot. successor tools

any tool yielding output
in mzML, featureXML
or consensusXML format

any tool that profits on reduced input

With this tool it is possible to extract m/z, retention time and intensity ranges from an input file and to write all data that lies within the given ranges to an output file.

Depending on the input file type, additional specific operations are possible:

mzML
- extract spectra of a certain MS level
- filter by signal-to-noise estimation
- filter by scan mode of the spectra
remove MS2 scans whose precursor matches identifications (from an idXML file in 'id:blacklist')
featureXML
- filter by feature charge
- filter by feature size (number of subordinate features)
- filter by overall feature quality
consensusXML
- filter by size (number of elements in consensus features)
- filter by consensus feature charge
- filter by map (extracts specified maps and re-evaluates consensus centroid)
  e.g. FileFilter -map 2 3 5 -in file1.consensusXML -out file2.consensusXML
  If a single map is specified, the feature itself can be extracted.
  e.g. FileFilter -map 5 -in file1.consensusXML -out file2.featureXML
featureXML / consensusXML:
remove items with a certain meta value annotation. Allowing for >, < and = comparisons. List types are compared by length, not content. Integer, Double and String are compared using their build-in operators.
- filter sequences, e.g. "LYSNLVER" or the modification "(Phospho)"
  e.g. FileFilter -id:sequences_whitelist Phospho -in file1.consensusXML -out file2.consensusXML
- filter accessions, e.g. "sp|P02662|CASA1_BOVIN"
- remove features with annotations
- remove features without annotations
- remove unassigned peptide identifications
- filter id with best score of features with multiple peptide identifications
  e.g. FileFilter -id:remove_unannotated_features -id:remove_unassigned_ids -id:keep_best_score_id -in file1.featureXML -out file2.featureXML
- remove features with id clashes (different sequences mapped to one feature)

The priority of the id-flags is (decreasing order): remove_annotated_features / remove_unannotated_features -> remove_clashes -> keep_best_score_id -> sequences_whitelist / accessions_whitelist

MS2 and higher spectra can be filtered according to precursor m/z (see 'pc_mz'). This flag can be combined with 'rt' range to filter precursors by RT and m/z. If you want to extract an MS1 region with untouched MS2 spectra included, you will need to split the dataset by MS level and use 'mz' option for MS1 and 'pc_mz' for MS2 data. Then merge them again. RT can be filtered at any step.

The command line parameters of this tool are:

FileFilter -- Extracts or manipulates portions of data from peak, feature or consensus-feature files.
Version: 1.10.0 Mar  7 2013, 08:30:22, Revision: unknown

Usage:
  FileFilter <options>

This tool has algoritm parameters which can only be used via an INI file and are not accessible from the command line!

Options (mandatory options marked with '*'):
  -in <file>*                                      Input file  (valid formats: 'mzML', 'featureXML', 'consens
                                                   usXML')
  -in_type <type>                                  Input file type -- default: determined from file extension
                                                   or content
                                                   (valid: 'mzML', 'featureXML', 'consensusXML')
  -out <file>*                                     Output file (valid formats: 'mzML', 'featureXML', 'consens
                                                   usXML')
  -out_type <type>                                 Output file type -- default: determined from file extensio
                                                   n or content
                                                   (valid: 'mzML', 'featureXML', 'consensusXML')
  -rt [min]:[max]                                  Retention time range to extract (default: ':')
  -mz [min]:[max]                                  M/z range to extract (applies to ALL ms levels!) (default:
                                                   ':')
  -pc_mz [min]:[max]                               MSn (n>=2) precursor filtering according to their m/z valu
                                                   e. Do not use this flag in conjunction with 'mz', unless
                                                   you want to actually remove peaks in spectra (see 'mz').
                                                   RT filtering is covered by 'rt' and compatible with this
                                                   flag. (default: ':')
  -int [min]:[max]                                 Intensity range to extract (default: ':')
  -sort                                            Sorts the output according to RT and m/z.

Peak data options:
  -peak_options:sn <s/n ratio>                     Write peaks with S/N > 'sn' values only (default: '0')
  -peak_options:rm_pc_charge i j ...               Remove MS(2) spectra with these precursor charges. All 
                                                   spectra without precursor are kept!
  -peak_options:level i j ...                      MS levels to extract (default: '[1 2 3]')
  -peak_options:sort_peaks                         Sorts the peaks according to m/z.
  -peak_options:no_chromatograms                   No conversion to space-saving real chromatograms, e.g. 
                                                   from SRM scans.
  -peak_options:remove_chromatograms               Removes chromatograms stored in a file.
  -peak_options:mz_precision 32 or 64              Store base64 encoded m/z data using 32 or 64 bit precision
                                                   . (default: '64' valid: '32', '64')
  -peak_options:int_precision 32 or 64             Store base64 encoded intensity data using 32 or 64 bit 
                                                   precision. (default: '32' valid: '32', '64')

Remove spectra or select spectra (removing all others) with certain properties.:
  -spectra:remove_zoom                             Remove zoom (enhanced resolution) scans
  -spectra:remove_mode <mode>                      Remove scans by scan mode
                                                   (valid: 'Unknown', 'MassSpectrum', 'MS1Spectrum', 'MSnSp
                                                   ectrum', 'SelectedIonMonitoring', 'SelectedReactionMonitor
                                                   ing', 'ConsecutiveReactionMonitoring', 'ConstantNeutralGai
                                                   n', 'ConstantNeutralLoss', 'Precursor', 'EnhancedMultiplyC
                                                   harged', 'TimeDelayedFragmentation', 'ElectromagneticRadia
                                                   tion', 'Emission', 'Absorbtion')

                                                   

Remove spectra or select spectra (removing all others) with certain properties.:
  -spectra:remove_activation <activation>          Remove MSn scans where any of its precursors features a 
                                                   certain activation method
                                                   (valid: 'Collision-induced dissociation', 'Post-source d
                                                   ecay', 'Plasma desorption', 'Surface-induced dissociation'
                                                   , 'Blackbody infrared radiative dissociation', 'Electron
                                                   capture dissociation', 'Infrared multiphoton dissociation'
                                                   , 'Sustained off-resonance irradiation', 'High-energy coll
                                                   ision-induced dissociation', 'Low-energy collision-induced
                                                   dissociation', 'Photodissociation', 'Electron transfer
                                                   dissociation', 'Pulsed q dissociation')

                                                   

Remove spectra or select spectra (removing all others) with certain properties.:
  -spectra:select_zoom                             Select zoom (enhanced resolution) scans
  -spectra:select_mode <mode>                      Selects scans by scan mode
                                                   (valid: 'Unknown', 'MassSpectrum', 'MS1Spectrum', 'MSnSp
                                                   ectrum', 'SelectedIonMonitoring', 'SelectedReactionMonitor
                                                   ing', 'ConsecutiveReactionMonitoring', 'ConstantNeutralGai
                                                   n', 'ConstantNeutralLoss', 'Precursor', 'EnhancedMultiplyC
                                                   harged', 'TimeDelayedFragmentation', 'ElectromagneticRadia
                                                   tion', 'Emission', 'Absorbtion')
  -spectra:select_activation <activation>          Select MSn scans where any of its precursors features a 
                                                   certain activation method
                                                   (valid: 'Collision-induced dissociation', 'Post-source d
                                                   ecay', 'Plasma desorption', 'Surface-induced dissociation'
                                                   , 'Blackbody infrared radiative dissociation', 'Electron
                                                   capture dissociation', 'Infrared multiphoton dissociation'
                                                   , 'Sustained off-resonance irradiation', 'High-energy coll
                                                   ision-induced dissociation', 'Low-energy collision-induced
                                                   dissociation', 'Photodissociation', 'Electron transfer
                                                   dissociation', 'Pulsed q dissociation')

                                                   

Feature data options:
  -feature:q [min]:[max]                           Overall quality range to extract [0:1] (default: ':')

                                                   

Consensus feature data options:
  -consensusfeature:map i j ...                    Maps to be extracted from a consensus
  -consensusfeature:map_and                        AND connective of map selection instead of OR.

                                                   

Feature & Consensus data options:
  -f_and_cf:charge [min]:[max]                     Charge range to extract (default: ':')
  -f_and_cf:size [min]:[max]                       Size range to extract (default: ':')
  -f_and_cf:remove_meta <name> 'lt|eq|gt' <value>  Expects a 3-tuple (=3 entries in the list), i.e. <name> 
                                                   'lt|eq|gt' <value>; the first is the name of meta value,
                                                   followed by the comparison operator (equal, less or greate
                                                   r) and the value to compare to. All comparisons are done
                                                   after converting the given value to the corresponding data
                                                   value type of the meta value (for lists, this simply comp
                                                   ares length, not content!)!

                                                   

ID options. The Priority of the id-flags is: remove_annotated_features / remove_unannotated_features -> remov
e_clashes -> keep_best_score_id -> sequences_whitelist / accessions_whitelist.:
  -id:keep_best_score_id                           In case of multiple peptide identifications, keep only 
                                                   the id with best score
  -id:sequences_whitelist <sequence>               Keep only features with white listed sequences, e.g. LYSNL
                                                   VER or the modification (Oxidation)
  -id:accessions_whitelist <accessions>            Keep only features with white listed accessions, e.g. sp|P
                                                   02662|CASA1_BOVIN
  -id:remove_annotated_features                    Remove features with annotations
  -id:remove_unannotated_features                  Remove features without annotations
  -id:remove_unassigned_ids                        Remove unassigned peptide identifications
  -id:blacklist <file>                             Input file containing MS2 identifications whose correspond
                                                   ing MS2 spectra should be removed from the mzML file!
                                                   Matching tolerances are taken from 'id:rt' and 'id:mz' op
                                                   tions.
                                                   This tool will require all IDs to be matched to an MS2 sp
                                                   ectrum, and quit with error otherwise. Use 'id:blacklist_i
                                                   mperfect' to allow for mismatches. (valid formats: 'idXML'
                                                   )
  -id:rt tolerance                                 Retention tolerance [s] for precursor to id position (defa
                                                   ult: '0.1' min: '0')
  -id:mz tolerance                                 M/z tolerance [Th] for precursor to id position (default: 
                                                   '0.001' min: '0')
  -id:blacklist_imperfect                          Allow for mismatching precursor positions (see 'id:blackli
                                                   st')

                                                   
                                                   
Common TOPP options:
  -ini <file>                                      Use the given TOPP INI file
  -threads <n>                                     Sets the number of threads allowed to be used by the TOPP 
                                                   tool (default: '1')
  -write_ini <file>                                Writes the default configuration file
  --help                                           Shows options
  --helphelp                                       Shows all options (including advanced)

The following configuration subsections are valid:
 - algorithm   S/N algorithm section

You can write an example INI file using the '-write_ini' option.
Documentation of subsection parameters can be found in the
doxygen documentation or the INIFileEditor.
Have a look at OpenMS documentation for more information.

INI file documentation of this tool:

Legend:

required parameter

advanced parameter

+FileFilterExtracts or manipulates portions of data from peak, feature or consensus-feature files.

version1.10.0 Version of the tool that generated this parameters file.

++1Instance '1' section for 'FileFilter'

in input file input file*.mzML,*.featureXML,*.consensusXML

in_type input file type -- default: determined from file extension or content
mzML,featureXML,consensusXML

out output fileoutput file*.mzML,*.featureXML,*.consensusXML

out_type output file type -- default: determined from file extension or content
mzML,featureXML,consensusXML

rt: retention time range to extract

mz: m/z range to extract (applies to ALL ms levels!)

pc_mz: MSn (n>=2) precursor filtering according to their m/z value. Do not use this flag in conjunction with 'mz', unless you want to actually remove peaks in spectra (see 'mz'). RT filtering is covered by 'rt' and compatible with this flag.

int: intensity range to extract

sortfalse sorts the output according to RT and m/z.true,false

log Name of log file (created only when specified)

debug0 Sets the debug level

threads1 Sets the number of threads allowed to be used by the TOPP tool

no_progressfalse Disables progress logging to command linetrue,false

testfalse Enables the test mode (needed for internal use only)true,false

+++peak_optionsPeak data options

sn0 write peaks with S/N > 'sn' values only

rm_pc_charge[] Remove MS(2) spectra with these precursor charges. All spectra without precursor are kept!

level[1, 2, 3] MS levels to extract

sort_peaksfalse sorts the peaks according to m/z.true,false

no_chromatogramsfalse No conversion to space-saving real chromatograms, e.g. from SRM scans.true,false

remove_chromatogramsfalse Removes chromatograms stored in a file.true,false

mz_precision64 Store base64 encoded m/z data using 32 or 64 bit precision.32,64

int_precision32 Store base64 encoded intensity data using 32 or 64 bit precision.32,64

+++spectraRemove spectra or select spectra (removing all others) with certain properties.

remove_zoomfalse Remove zoom (enhanced resolution) scanstrue,false

remove_mode Remove scans by scan mode
Unknown,MassSpectrum,MS1Spectrum,MSnSpectrum,SelectedIonMonitoring,SelectedReactionMonitoring,ConsecutiveReactionMonitoring,ConstantNeutralGain,ConstantNeutralLoss,Precursor,EnhancedMultiplyCharged,TimeDelayedFragmentation,ElectromagneticRadiation,Emission,Absorbtion

remove_activation Remove MSn scans where any of its precursors features a certain activation method
Collision-induced dissociation,Post-source decay,Plasma desorption,Surface-induced dissociation,Blackbody infrared radiative dissociation,Electron capture dissociation,Infrared multiphoton dissociation,Sustained off-resonance irradiation,High-energy collision-induced dissociation,Low-energy collision-induced dissociation,Photodissociation,Electron transfer dissociation,Pulsed q dissociation

select_zoomfalse Select zoom (enhanced resolution) scanstrue,false

select_mode Selects scans by scan mode
Unknown,MassSpectrum,MS1Spectrum,MSnSpectrum,SelectedIonMonitoring,SelectedReactionMonitoring,ConsecutiveReactionMonitoring,ConstantNeutralGain,ConstantNeutralLoss,Precursor,EnhancedMultiplyCharged,TimeDelayedFragmentation,ElectromagneticRadiation,Emission,Absorbtion

select_activation Select MSn scans where any of its precursors features a certain activation method
Collision-induced dissociation,Post-source decay,Plasma desorption,Surface-induced dissociation,Blackbody infrared radiative dissociation,Electron capture dissociation,Infrared multiphoton dissociation,Sustained off-resonance irradiation,High-energy collision-induced dissociation,Low-energy collision-induced dissociation,Photodissociation,Electron transfer dissociation,Pulsed q dissociation

+++featureFeature data options

q: Overall quality range to extract [0:1]

+++consensusfeatureConsensus feature data options

map[] maps to be extracted from a consensus

map_andfalse AND connective of map selection instead of OR.true,false

+++f_and_cfFeature & Consensus data options

charge: charge range to extract

size: size range to extract

remove_meta[] Expects a 3-tuple (=3 entries in the list), i.e. 'lt|eq|gt' ; the first is the name of meta value, followed by the comparison operator (equal, less or greater) and the value to compare to. All comparisons are done after converting the given value to the corresponding data value type of the meta value (for lists, this simply compares length, not content!)!

+++idID options. The Priority of the id-flags is: remove_annotated_features / remove_unannotated_features -> remove_clashes -> keep_best_score_id -> sequences_whitelist / accessions_whitelist.

remove_clashesfalse remove features with id clashes (different sequences mapped to one feature)true,false

keep_best_score_idfalse in case of multiple peptide identifications, keep only the id with best scoretrue,false

sequences_whitelist[] keep only features with white listed sequences, e.g. LYSNLVER or the modification (Oxidation)

accessions_whitelist[] keep only features with white listed accessions, e.g. sp|P02662|CASA1_BOVIN

remove_annotated_featuresfalse remove features with annotationstrue,false

remove_unannotated_featuresfalse remove features without annotationstrue,false

remove_unassigned_idsfalse remove unassigned peptide identificationstrue,false

blacklist Input file containing MS2 identifications whose corresponding MS2 spectra should be removed from the mzML file!
Matching tolerances are taken from 'id:rt' and 'id:mz' options.
This tool will require all IDs to be matched to an MS2 spectrum, and quit with error otherwise. Use 'id:blacklist_imperfect' to allow for mismatches.input file*.idXML

rt0.1 retention tolerance [s] for precursor to id position0:∞

mz0.001 m/z tolerance [Th] for precursor to id position0:∞

blacklist_imperfectfalse Allow for mismatching precursor positions (see 'id:blacklist')true,false

+++algorithmS/N algorithm section

++++SignalToNoise

max_intensity-1 maximal intensity considered for histogram construction. By default, it will be calculated automatically (see auto_mode). Only provide this parameter if you know what you are doing (and change 'auto_mode' to '-1')! All intensities EQUAL/ABOVE 'max_intensity' will be added to the LAST histogram bin. If you choose 'max_intensity' too small, the noise estimate might be too small as well. If chosen too big, the bins become quite large (which you could counter by increasing 'bin_count', which increases runtime). In general, the Median-S/N estimator is more robust to a manual max_intensity than the MeanIterative-S/N.-1:∞

auto_max_stdev_factor3 parameter for 'max_intensity' estimation (if 'auto_mode' == 0): mean + 'auto_max_stdev_factor' * stdev0:999

auto_max_percentile95 parameter for 'max_intensity' estimation (if 'auto_mode' == 1): auto_max_percentile th percentile0:100

auto_mode0 method to use to determine maximal intensity: -1 --> use 'max_intensity'; 0 --> 'auto_max_stdev_factor' method (default); 1 --> 'auto_max_percentile' method-1:1

win_len200 window length in Thomson1:∞

bin_count30 number of bins for intensity values3:∞

min_required_elements10 minimum number of elements required in a window (otherwise it is considered sparse)1:∞

noise_for_empty_window1e+20 noise value used for sparse windows

For the parameters of the S/N algorithm section see the class documentation there:
peak_options:sn

Todo:: add tests for selecting modes (port remove modes) (Andreas)