biolockj.module.implicit

Class Demultiplexer

java.lang.Object

biolockj.module.BioModuleImpl

biolockj.module.ScriptModuleImpl

biolockj.module.JavaModuleImpl

biolockj.module.implicit.Demultiplexer

All Implemented Interfaces:

BioModule, JavaModule, ScriptModule, SeqModule, Comparable<BioModule>

public class Demultiplexer
extends JavaModuleImpl
implements SeqModule

This BioModule splits multiplexed data into a separate file or pair of files (for paired reads) for each sample. The sample ID, or an identification barcode must be included in the header of each read. Barcodes are matched to Sample IDs using the metadata column configured via property "metadata.barcodeColumn".
If demultiplexer.strategy=barcode_in_header, register a match if the barcode is found anywhere in the sequence header.
If demultiplexer.strategy=barcode_in_seq, register a match if the sequence itself starts with any of the barcodes. The output multiplexed file will have barcodes removed from the sequences since they no longer serve any purpose.
Sequences with no matching barcode, or without a matching paired header (if paired), are output to a "NO_MATCH" file in the module temp directory.
Paired reads are accepted in 2 multiplexed formats:

1. Forward and reverse reads can be partitioned into 2 separate files identified by identified by: input.suffixFw and input.suffixRv.
2. Forward and reverse reads can be combined in a single file with read direction indicated in the header by values "1:N:" and "2:N:"

GUI Module Name

Demultiplexer

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	NUM_LINES_TEMP_FILE Module splits multiplexed file into smaller files with this number of lines: 2000000
protected static String	SAMPLE_ID_SUFFIX_TRIM_DEFAULT Multiplexed files created by BioLockJ may add sample ID to the sequence header if no barcode is provided. If sample ID is added, it is immediately followed by the character: "_" This value can be used then to set Config."input.trimSuffix"

Fields inherited from class biolockj.module.JavaModuleImpl

BLJ_OPTIONS

Fields inherited from class biolockj.module.BioModuleImpl

GZIP_EXT, LOG_EXT, PDF_EXT, RETURN, SH_EXT, TAB_DELIM, TSV_EXT, TXT_EXT

Fields inherited from interface biolockj.module.ScriptModule

SCRIPT_BATCH_SIZE, SCRIPT_DEFAULT_HEADER, SCRIPT_NUM_THREADS, SCRIPT_PERMISSIONS, SCRIPT_TIMEOUT

Fields inherited from interface biolockj.module.BioModule

MAIN_SCRIPT_PREFIX, OUTPUT_DIR, TEMP_DIR

Constructor Summary

Constructors

Constructor and Description
Demultiplexer()

Method Summary

Modifier and Type	Method and Description
protected void	breakUpFiles() Some multiplexed files can be very large.
void	checkDependencies() Validate module dependencies: If Config."demultiplexer.strategy" indicates use of barcodes to demultiplexer, validate metadata column named Config."metadata.barcodeColumn" exists Call setMultiplexedConfig() to set multiplexed Config if needed If Config."demultiplexer.barcodeCutoff" defined, validate between 0.0 - 1.0
void	cleanUp() Update SeqUtil to indicate data has been demultiplexed.
protected void	demultiplex(Map<String,Set<String>> validHeaders) Demultiplex the file into separate small temp files, with 2000000 lines each for processing.
List<File>	getSeqFiles(Collection<File> files) Return only sequence files for sample IDs found in the metadata file. If Config."metadata.required" = "Y", an error is thrown to list the files that cannot be matched to a metadata row.
String	getSummary() Produces initial count and demultiplexed output count summaries for forward/reverse reads.
protected Map<String,Set<String>>	getValidFwHeaders() Get valid forward read headers that belong to reads with a valid barcode or sample identifier.
protected Map<String,Set<String>>	getValidHeaders() This method obtains all valid headers for the forward reads, and returns only headers that also have a matching reverse read
void	runModule() Module execution summary: Execute breakUpFiles() to split the multiplex file into smaller size for processing Execute getValidHeaders() to obtain list of valid headers matched to sample ID with the metadata file and also verifies matching forward and reverse read headers if demuliplexing paired reads.
protected void	setMultiplexedConfig() Set the Config properties needed to read the sample IDs from a multiplexed file if no barcode is provided Set Config."input.trimPrefix" = 1st sequence header character. Set Config."input.trimSuffix" ="_"

Methods inherited from class biolockj.module.JavaModuleImpl

buildScript, executeTask, getSource, getWorkerScriptFunctions, isValidInputModule, markStatus, moduleComplete, moduleFailed

Methods inherited from class biolockj.module.ScriptModuleImpl

buildScriptForPairedReads, getJobParams, getMainScript, getRuntimeParams, getScriptDir, getScriptErrors, getTimeout, hasScripts

Methods inherited from class biolockj.module.BioModuleImpl

cacheInputFiles, compareTo, equals, findModuleInputFiles, getFileCache, getID, getInputFiles, getModuleDir, getOutputDir, getPostRequisiteModules, getPreRequisiteModules, getTempDir, init, toString, validateFileNameUnique

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface biolockj.module.ScriptModule

buildScript, buildScriptForPairedReads, getJobParams, getMainScript, getScriptDir, getScriptErrors, getTimeout, getWorkerScriptFunctions

Methods inherited from interface biolockj.module.BioModule

executeTask, getID, getInputFiles, getModuleDir, getOutputDir, getPostRequisiteModules, getPreRequisiteModules, getTempDir, init, isValidInputModule

Field Detail

NUM_LINES_TEMP_FILE

protected static final int NUM_LINES_TEMP_FILE

Module splits multiplexed file into smaller files with this number of lines: 2000000

See Also:

Constant Field Values

SAMPLE_ID_SUFFIX_TRIM_DEFAULT

protected static final String SAMPLE_ID_SUFFIX_TRIM_DEFAULT

Multiplexed files created by BioLockJ may add sample ID to the sequence header if no barcode is provided.
If sample ID is added, it is immediately followed by the character: "_"
This value can be used then to set Config."input.trimSuffix"

See Also:

Constant Field Values

Constructor Detail

Demultiplexer

public Demultiplexer()

Method Detail

checkDependencies

public void checkDependencies()
                       throws Exception

Validate module dependencies:

1. If Config."demultiplexer.strategy" indicates use of barcodes to demultiplexer, validate metadata column named Config."metadata.barcodeColumn" exists
2. Call setMultiplexedConfig() to set multiplexed Config if needed
3. If Config."demultiplexer.barcodeCutoff" defined, validate between 0.0 - 1.0

Specified by:

checkDependencies in interface BioModule

Overrides:

checkDependencies in class ScriptModuleImpl

Throws:

Exception - thrown if missing or invalid dependencies are found

cleanUp

public void cleanUp()
             throws Exception

Update SeqUtil to indicate data has been demultiplexed.

Specified by:

cleanUp in interface BioModule

Overrides:

cleanUp in class BioModuleImpl

Throws:

Exception - if unable to modify property

getSeqFiles

public List<File> getSeqFiles(Collection<File> files)
                       throws Exception

Description copied from interface: SeqModule

Return only sequence files for sample IDs found in the metadata file.
If Config."metadata.required" = "Y", an error is thrown to list the files that cannot be matched to a metadata row.

Specified by:

getSeqFiles in interface SeqModule

Parameters:

files - Module input files

Returns:

Module sequence files

Throws:

Exception - if no input files are found

getSummary

public String getSummary()
                  throws Exception

Produces initial count and demultiplexed output count summaries for forward/reverse reads.

Specified by:

getSummary in interface BioModule

Overrides:

getSummary in class ScriptModuleImpl

Returns:

Summary of BioModule execution

Throws:

Exception - if any error occurs

runModule

public void runModule()
               throws Exception

Module execution summary:

1. Execute breakUpFiles() to split the multiplex file into smaller size for processing
2. Execute getValidHeaders() to obtain list of valid headers matched to sample ID with the metadata file and also verifies matching forward and reverse read headers if demuliplexing paired reads.
3. Execute demultiplex(Map) to demultiplex the data into a separate file (or pair of files) for each sample

If paired reads are combined in a single file the read direction must be identified in the sequence header using key strings " 1:N:" " 2:N:"

Specified by:

runModule in interface JavaModule

Specified by:

runModule in class JavaModuleImpl

Throws:

Exception - thrown if any runtime error occurs

breakUpFiles

protected void breakUpFiles()
                     throws Exception

Some multiplexed files can be very large. This method breaks each input file into smaller files, each 2000000 lines in size to avoid memory issues while processing.

Throws:

Exception - if unexpected errors occur at runtime

demultiplex

protected void demultiplex(Map<String,Set<String>> validHeaders)
                    throws Exception

Demultiplex the file into separate small temp files, with 2000000 lines each for processing. If paired reads, generate separate forward and reverse files for each sample output file named by sample ID.

Parameters:

validHeaders - Set of valid headers

Throws:

Exception - if error occurs reading the multiplexed file

getValidFwHeaders

protected Map<String,Set<String>> getValidFwHeaders()
                                             throws Exception

Get valid forward read headers that belong to reads with a valid barcode or sample identifier.

Returns:

Map sampleID to list of valid headers

Throws:

Exception - if error occur

getValidHeaders

protected Map<String,Set<String>> getValidHeaders()
                                           throws Exception

This method obtains all valid headers for the forward reads, and returns only headers that also have a matching reverse read

Returns:

Map of valid headers for each sample ID

Throws:

Exception - if unable to obtain headers

setMultiplexedConfig

protected void setMultiplexedConfig()
                             throws Exception

Set the Config properties needed to read the sample IDs from a multiplexed file if no barcode is provided
Set Config."input.trimPrefix" = 1st sequence header character.
Set Config."input.trimSuffix" ="_"

Throws:

Exception - if unable to update the property values