asap.pointmatch package

Submodules

asap.pointmatch.create_tilepairs module

class asap.pointmatch.create_tilepairs.TilePairClientModule(schema_type=None, *args, **kwargs)

Bases: RenderModule

Note

This class takes a ArgSchema as an input to parse inputs , with a default schema of type TilePairClientParameters

client_class = 'org.janelia.render.client.TilePairClient'

client_script_name = 'run_ws_client.sh'

default_output_schema

alias of TilePairClientOutputParameters

default_schema

alias of TilePairClientParameters

run()

asap.pointmatch.generate_point_matches_qsub module

class asap.pointmatch.generate_point_matches_qsub.PointMatchClientModuleQsub(schema_type=None, *args, **kwargs)

Bases: RenderModule

Note

This class takes a ArgSchema as an input to parse inputs , with a default schema of type RenderParameters

run()

asap.pointmatch.generate_point_matches_spark module

class asap.pointmatch.generate_point_matches_spark.PointMatchClientModuleSpark(input_data=None, schema_type=None, output_schema_type=None, args=None, logger_name='argschema.argschema_parser')

Bases: SparkModule

Note

This class takes a ArgSchema as an input to parse inputs , with a default schema of type PointMatchClientParametersSpark

default_output_schema

alias of PointMatchClientOutputSchema

default_schema

alias of PointMatchClientParametersSpark

classmethod get_args(**kwargs)

override to append to spark call

classmethod get_pointmatch_args(baseDataUrl=None, owner=None, collection=None, pairJson=None, SIFTfdSize=None, SIFTminScale=None, SIFTmaxScale=None, SIFTsteps=None, matchRod=None, matchModelType=None, matchIterations=None, matchMaxEpsilon=None, matchMinInlierRatio=None, matchMinNumInliers=None, matchMaxNumInliers=None, matchMaxTrust=None, maxFeatureCacheGb=None, clipWidth=None, clipHeight=None, renderScale=None, renderWithFilter=None, renderWithoutMask=None, renderFullScaleWidth=None, renderFullScaleHeight=None, fillWithNoise=None, rootFeatureDirectory=None, renderFilterListName=None, requireStoredFeatures=None, matchFilter=None, **kwargs)

run()

asap.pointmatch.generate_point_matches_spark.add_arg(l, argname, args)

asap.pointmatch.generate_point_matches_spark.form_sift_params_list(args)

asap.pointmatch.generate_point_matches_spark.get_host_port_dict_from_url(url)

asap.pointmatch.schemas module

class asap.pointmatch.schemas.CollectionId(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: Schema

CollectionId

key	description	default	field_type	json_type
owner	owner of collection	(REQUIRED)	String	str
name	name of collection	(REQUIRED)	String	str

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.PointMatchClientOutputSchema(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: Schema

PointMatchClientOutputSchema

key	description	default	field_type	json_type
collectionId	collection identifying details	(REQUIRED)	CollectionId	dict
pairCount	number of tile pairs in collection	(REQUIRED)	Integer	int

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.PointMatchClientParametersQsub(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: RenderParameters, SIFTPointMatchParameters, SparkOptions

This schema is designed to be a schema_type for an ArgSchemaParser object

PointMatchClientParametersQsub

key	description	default	field_type	json_type
jarfile	spark jar to call java spark command	(REQUIRED)	String	str
className	spark class to call	(REQUIRED)	String	str
driverMemory	spark driver memory (important for local spark)	6g	String	str
memory	Memory required for spark job	NA	String	str
sparkhome	Path to the spark home directory	/allen/aibs/pipeline/image_processing/volume_assembly/utils/spark	InputDir	str
spark_files	list of spark files to add to the spark submit command	NA	List	str
spark_conf	dictionary of key value pairs to add to spark_submit as –conf key=value	NA	Dict	?
baseDataUrl	api endpoint url e.g. http://<host>[:port]/render-ws/v1	NA	String	str
owner	owner of match collection	NA	String	str
collection	match collection name	NA	String	str
render	parameters to connect to render server	(REQUIRED)	RenderClientParameters	dict
matchRod	Ratio of first to second nearest neighbors used as a cutoff in matching features. 0.92 if excluded or None	NA	Float	float
matchModelType	Model to match for RANSAC filtering. ‘AFFINE’ if excluded or None	NA	String	str
matchIterations	RANSAC filter iterations. 1000 if excluded or None	NA	Integer	int
matchMaxEpsilon	no description	NA	Float	float
matchMinInlierRatio	Minimal ratio of inliers to candidates for successful RANSAC filtering. 0.0 if excluded or None	NA	Float	float
matchMinNumInliers	Minimum absolute number of inliers for successful RANSAC filtering. 4 if excluded or None	NA	Integer	int
matchMaxNumInliers	Maximum absolute number of inliers allowed after RANSAC filtering. unlimited if excluded or None	NA	Integer	int
matchMaxTrust	Maximum trust for filtering such that candidates with cost larger than matchMaxTrust * median cost are rejected. 3.0 if excluded or None	NA	Float	float
matchFilter	whether to match one set of matches, or multiple sets. And, whether to keep them separate, or aggregate them. SINGLE_SET if excluded or None.	NA	String	str
rootFeatureDirectory	Root directory for saved feature lists. Features extracted from dynamically rendered canvases if excluded or None.	NA	String	str
requireStoredFeatures	Whether to throw an exception in case features stored in rootFeatureDirectory cannot be found. Missing features are extracted from dynamically rendered canvases if excluded or None	NA	Boolean	bool
maxFeatureCacheGb	Maximum size of feature cache, in GB. 2GB if excluded or None	NA	Integer	int
clipWidth	Full scale pixels to include in clipped rendering of LEFT/RIGHT oriented tile pairs. Will not LEFT/RIGHT clip if excluded or None.	NA	Integer	int
clipHeight	Full scale pixels to include in clipped rendering of TOP/BOTTOM oriented tile pairs. Will not TOP/BOTTOM clip if excluded or None.	NA	Integer	int
renderScale	Scale at which image tiles will be rendered. 1.0 (full scale) if excluded or None	NA	Float	float
renderWithFilter	Render tiles using default filtering (0 and 255 pixel values replaced with integer in U(64, 191), followed by default NormalizeLocalContrast). True if excluded or None	NA	Boolean	bool
renderWithoutMask	Render tiles without mipMapLevel masks. True if excluded or None	NA	Boolean	bool
renderFullScaleWidth	Full scale width for all rendered tiles	NA	Integer	int
renderFullScaleHeight	Full scale height for all rendered tiles	NA	Integer	int
fillWithNoise	Fill each canvas image with noise prior to rendering. True if excluded or None	NA	Boolean	bool
renderFilterListName	Apply specified filter list to all renderings	NA	String	str
SIFTfdSize	SIFT feature descriptor size – samples per row and column. 8 if excluded or None	NA	Integer	int
SIFTminScale	SIFT minimum scale – minSize * minScale < size < maxSize * maxScale. 0.5 if excluded or None	NA	Float	float
SIFTmaxScale	SIFT maximum scale – minSize * minScale < size < maxSize * maxScale. 0.85 if excluded or None	NA	Float	float
SIFTsteps	SIFT steps per scale octave. 3 if excluded or None	NA	Integer	int
input_json	file path of input json file	NA	InputFile	str
output_json	file path to output json file	NA	OutputFile	str
log_level	set the logging level of the module	ERROR	LogLevel	str
pairJson	JSON file where tile pairs are stored (.json, .gz, .zip)	(REQUIRED)	InputFile	str
pbs_template	pbs template to wrap spark job	(REQUIRED)	InputFile	str
no_nodes	Number of nodes to run the pbs job	30	Integer	int
ppn	Number of processors per node (default = 30)	30	Integer	int
queue_name	Name of the queue to submit the job	connectome	String	str
logdir	location to set logging for qsub command	(REQUIRED)	OutputDir	str

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.PointMatchClientParametersSpark(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: SparkParameters, SIFTPointMatchParameters

This schema is designed to be a schema_type for an ArgSchemaParser object

PointMatchClientParametersSpark

key	description	default	field_type	json_type
baseDataUrl	api endpoint url e.g. http://<host>[:port]/render-ws/v1	NA	String	str
owner	owner of match collection	NA	String	str
collection	match collection name	NA	String	str
render	no description	NA	RenderClientParameters	dict
matchRod	Ratio of first to second nearest neighbors used as a cutoff in matching features. 0.92 if excluded or None	NA	Float	float
matchModelType	Model to match for RANSAC filtering. ‘AFFINE’ if excluded or None	NA	String	str
matchIterations	RANSAC filter iterations. 1000 if excluded or None	NA	Integer	int
matchMaxEpsilon	no description	NA	Float	float
matchMinInlierRatio	Minimal ratio of inliers to candidates for successful RANSAC filtering. 0.0 if excluded or None	NA	Float	float
matchMinNumInliers	Minimum absolute number of inliers for successful RANSAC filtering. 4 if excluded or None	NA	Integer	int
matchMaxNumInliers	Maximum absolute number of inliers allowed after RANSAC filtering. unlimited if excluded or None	NA	Integer	int
matchMaxTrust	Maximum trust for filtering such that candidates with cost larger than matchMaxTrust * median cost are rejected. 3.0 if excluded or None	NA	Float	float
matchFilter	whether to match one set of matches, or multiple sets. And, whether to keep them separate, or aggregate them. SINGLE_SET if excluded or None.	NA	String	str
rootFeatureDirectory	Root directory for saved feature lists. Features extracted from dynamically rendered canvases if excluded or None.	NA	String	str
requireStoredFeatures	Whether to throw an exception in case features stored in rootFeatureDirectory cannot be found. Missing features are extracted from dynamically rendered canvases if excluded or None	NA	Boolean	bool
maxFeatureCacheGb	Maximum size of feature cache, in GB. 2GB if excluded or None	NA	Integer	int
clipWidth	Full scale pixels to include in clipped rendering of LEFT/RIGHT oriented tile pairs. Will not LEFT/RIGHT clip if excluded or None.	NA	Integer	int
clipHeight	Full scale pixels to include in clipped rendering of TOP/BOTTOM oriented tile pairs. Will not TOP/BOTTOM clip if excluded or None.	NA	Integer	int
renderScale	Scale at which image tiles will be rendered. 1.0 (full scale) if excluded or None	NA	Float	float
renderWithFilter	Render tiles using default filtering (0 and 255 pixel values replaced with integer in U(64, 191), followed by default NormalizeLocalContrast). True if excluded or None	NA	Boolean	bool
renderWithoutMask	Render tiles without mipMapLevel masks. True if excluded or None	NA	Boolean	bool
renderFullScaleWidth	Full scale width for all rendered tiles	NA	Integer	int
renderFullScaleHeight	Full scale height for all rendered tiles	NA	Integer	int
fillWithNoise	Fill each canvas image with noise prior to rendering. True if excluded or None	NA	Boolean	bool
renderFilterListName	Apply specified filter list to all renderings	NA	String	str
SIFTfdSize	SIFT feature descriptor size – samples per row and column. 8 if excluded or None	NA	Integer	int
SIFTminScale	SIFT minimum scale – minSize * minScale < size < maxSize * maxScale. 0.5 if excluded or None	NA	Float	float
SIFTmaxScale	SIFT maximum scale – minSize * minScale < size < maxSize * maxScale. 0.85 if excluded or None	NA	Float	float
SIFTsteps	SIFT steps per scale octave. 3 if excluded or None	NA	Integer	int
input_json	file path of input json file	NA	InputFile	str
output_json	file path to output json file	NA	OutputFile	str
log_level	set the logging level of the module	ERROR	LogLevel	str
pairJson	JSON file where tile pairs are stored (.json, .gz, .zip)	(REQUIRED)	InputFile	str
jarfile	spark jar to call java spark command	(REQUIRED)	String	str
className	spark class to call	(REQUIRED)	String	str
driverMemory	spark driver memory (important for local spark)	6g	String	str
memory	Memory required for spark job	NA	String	str
sparkhome	Spark home directory containing bin/spark_submit	(REQUIRED)	InputDir	str
spark_files	list of spark files to add to the spark submit command	NA	List	str
spark_conf	dictionary of key value pairs to add to spark_submit as –conf key=value	NA	Dict	?
masterUrl	spark master url. For local execution local[num_procs,num_retries]	(REQUIRED)	String	str

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.PointMatchOpenCVParameters(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: RenderParameters

This schema is designed to be a schema_type for an ArgSchemaParser object

PointMatchOpenCVParameters

key	description	default	field_type	json_type
input_json	file path of input json file	NA	InputFile	str
output_json	file path to output json file	NA	OutputFile	str
log_level	set the logging level of the module	ERROR	LogLevel	str
render	parameters to connect to render server	(REQUIRED)	RenderClientParameters	dict
ndiv	one tile per tile pair subdivided into ndiv x ndiv for easier homography finding	8	Integer	int
matchMax	per tile pair limit, randomly chosen after SIFT and RANSAC	1000	Integer	int
downsample_scale	passed to cv2.resize(fx=, fy=)	0.3	Float	float
SIFT_nfeature	passed to cv2.xfeatures2d.SIFT_create(nfeatures=)	20000	Integer	int
SIFT_noctave	passed to cv2.xfeatures2d.SIFT_create(nOctaveLayers=)	3	Integer	int
SIFT_sigma	passed to cv2.xfeatures2d.SIFT_create(sigma=)	1.5	Float	float
RANSAC_outlier	passed to cv2.findHomography(src, dst, cv2.RANSAC, outlier)	5.0	Float	float
FLANN_ntree	passed to cv2.FlannBasedMatcher()	5	Integer	int
FLANN_ncheck	passed to cv2.FlannBasedMatcher()	50	Integer	int
ratio_of_dist	ratio in Lowe’s ratio test	0.7	Float	float
CLAHE_grid	tileGridSize for cv2 CLAHE	None	Integer	int
CLAHE_clip	clipLimit for cv2 CLAHE	None	Float	float
pairJson	full path of tilepair json	NA	String	str
input_stack	Name of raw input lens data stack	NA	String	str
match_collection	name of point match collection	NA	String	str
ncpus	number of CPUs to use	-1	Integer	int

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.SIFTPointMatchParameters(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: ArgSchema, FeatureExtractionParameters, FeatureRenderParameters, FeatureRenderClipParameters, FeatureStorageParameters, MatchDerivationParameters, RenderParametersMatchWebServiceParameters

This schema is designed to be a schema_type for an ArgSchemaParser object

SIFTPointMatchParameters

key	description	default	field_type	json_type
baseDataUrl	api endpoint url e.g. http://<host>[:port]/render-ws/v1	NA	String	str
owner	owner of match collection	NA	String	str
collection	match collection name	NA	String	str
render	no description	NA	RenderClientParameters	dict
matchRod	Ratio of first to second nearest neighbors used as a cutoff in matching features. 0.92 if excluded or None	NA	Float	float
matchModelType	Model to match for RANSAC filtering. ‘AFFINE’ if excluded or None	NA	String	str
matchIterations	RANSAC filter iterations. 1000 if excluded or None	NA	Integer	int
matchMaxEpsilon	no description	NA	Float	float
matchMinInlierRatio	Minimal ratio of inliers to candidates for successful RANSAC filtering. 0.0 if excluded or None	NA	Float	float
matchMinNumInliers	Minimum absolute number of inliers for successful RANSAC filtering. 4 if excluded or None	NA	Integer	int
matchMaxNumInliers	Maximum absolute number of inliers allowed after RANSAC filtering. unlimited if excluded or None	NA	Integer	int
matchMaxTrust	Maximum trust for filtering such that candidates with cost larger than matchMaxTrust * median cost are rejected. 3.0 if excluded or None	NA	Float	float
matchFilter	whether to match one set of matches, or multiple sets. And, whether to keep them separate, or aggregate them. SINGLE_SET if excluded or None.	NA	String	str
rootFeatureDirectory	Root directory for saved feature lists. Features extracted from dynamically rendered canvases if excluded or None.	NA	String	str
requireStoredFeatures	Whether to throw an exception in case features stored in rootFeatureDirectory cannot be found. Missing features are extracted from dynamically rendered canvases if excluded or None	NA	Boolean	bool
maxFeatureCacheGb	Maximum size of feature cache, in GB. 2GB if excluded or None	NA	Integer	int
clipWidth	Full scale pixels to include in clipped rendering of LEFT/RIGHT oriented tile pairs. Will not LEFT/RIGHT clip if excluded or None.	NA	Integer	int
clipHeight	Full scale pixels to include in clipped rendering of TOP/BOTTOM oriented tile pairs. Will not TOP/BOTTOM clip if excluded or None.	NA	Integer	int
renderScale	Scale at which image tiles will be rendered. 1.0 (full scale) if excluded or None	NA	Float	float
renderWithFilter	Render tiles using default filtering (0 and 255 pixel values replaced with integer in U(64, 191), followed by default NormalizeLocalContrast). True if excluded or None	NA	Boolean	bool
renderWithoutMask	Render tiles without mipMapLevel masks. True if excluded or None	NA	Boolean	bool
renderFullScaleWidth	Full scale width for all rendered tiles	NA	Integer	int
renderFullScaleHeight	Full scale height for all rendered tiles	NA	Integer	int
fillWithNoise	Fill each canvas image with noise prior to rendering. True if excluded or None	NA	Boolean	bool
renderFilterListName	Apply specified filter list to all renderings	NA	String	str
SIFTfdSize	SIFT feature descriptor size – samples per row and column. 8 if excluded or None	NA	Integer	int
SIFTminScale	SIFT minimum scale – minSize * minScale < size < maxSize * maxScale. 0.5 if excluded or None	NA	Float	float
SIFTmaxScale	SIFT maximum scale – minSize * minScale < size < maxSize * maxScale. 0.85 if excluded or None	NA	Float	float
SIFTsteps	SIFT steps per scale octave. 3 if excluded or None	NA	Integer	int
input_json	file path of input json file	NA	InputFile	str
output_json	file path to output json file	NA	OutputFile	str
log_level	set the logging level of the module	ERROR	LogLevel	str
pairJson	JSON file where tile pairs are stored (.json, .gz, .zip)	(REQUIRED)	InputFile	str

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.SwapPointMatches(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: RenderParameters

This schema is designed to be a schema_type for an ArgSchemaParser object

SwapPointMatches

key	description	default	field_type	json_type
input_json	file path of input json file	NA	InputFile	str
output_json	file path to output json file	NA	OutputFile	str
log_level	set the logging level of the module	ERROR	LogLevel	str
render	parameters to connect to render server	(REQUIRED)	RenderClientParameters	dict
match_owner	Match collection owner name	(REQUIRED)	String	str
source_collection	Source point match collection	(REQUIRED)	String	str
target_collection	Target point match collection	(REQUIRED)	String	str
zValues	List of integer group ids	(REQUIRED)	List	int
pool_size	Pool size	5	Integer	int

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.SwapPointMatchesOutput(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: DefaultSchema

SwapPointMatchesOutput

key	description	default	field_type	json_type
source_collection	Source point match collection	(REQUIRED)	String	str
target_collection	Target point match collection	(REQUIRED)	String	str
swapped_zs	List of group ids that got swapped	(REQUIRED)	List	int
nonswapped_zs	List of group ids that did not get swapped	(REQUIRED)	List	int

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.TilePairClientOutputParameters(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: DefaultSchema

TilePairClientOutputParameters

key	description	default	field_type	json_type
tile_pair_file	location of json file with tile pair inputs	(REQUIRED)	InputFile	str

opts = <marshmallow.schema.SchemaOpts object>

class asap.pointmatch.schemas.TilePairClientParameters(extra=None, only=None, exclude=(), prefix='', strict=None, many=False, context=None, load_only=(), dump_only=(), partial=False)

Bases: RenderParameters

This schema is designed to be a schema_type for an ArgSchemaParser object

TilePairClientParameters

key	description	default	field_type	json_type
input_json	file path of input json file	NA	InputFile	str
output_json	file path to output json file	NA	OutputFile	str
log_level	set the logging level of the module	ERROR	LogLevel	str
render	parameters to connect to render server	(REQUIRED)	RenderClientParameters	dict
stack	input stack to which tilepairs need to be generated	(REQUIRED)	String	str
baseStack	Base stack	None	String	str
minZ	z min for generating tilepairs	None	Integer	int
maxZ	z max for generating tilepairs	None	Integer	int
xyNeighborFactor	Multiply this by max(width, height) of each tile to determine radius for locating neighbor tiles	0.9	Float	float
zNeighborDistance	Look for neighbor tiles with z values less than or equal to this distance from the current tile’s z value	2	Integer	int
excludeCornerNeighbors	Exclude neighbor tiles whose center x and y is outside the source tile’s x and y range respectively	True	Boolean	bool
excludeSameLayerNeighbors	Exclude neighbor tiles in the same layer (z) as the source tile	False	Boolean	bool
excludeCompletelyObscuredTiles	Exclude tiles that are completely obscured by reacquired tiles	True	Boolean	bool
output_dir	Output directory path to save the tilepair json file	(REQUIRED)	OutputDir	str
memGB	Memory for the java client to run	6G	String	str

opts = <marshmallow.schema.SchemaOpts object>

validate_data(data)

Module contents