tlve

1 Overview

The tlve program is a command-line tool for parsing different tlv (tag-length-value) structures and for printing them in various text-based formats. In order to correctly parse the input data tlve uses a text-based configuration. Default configuration file is ~/.tlverc (tlve.rc in windows).

tlve can parse text or binary (including ASN.1 BER) tlv structures. Tag, length and value can have fixed length or separated format. Values can be presented in different formats and converted to different encoding.

tlve was written by Timo Savinen.

This file documents version 2.1 of tlve.

2 Sample Output

Here are some realistic examples of running tlve.

This is the output of the command ‘cat ber.rc’: 

     $cat ber.rc
     # ASN.1 ber example configuration.
     # This file is for reading any ASN.1 BER encoded file, this
     # example contains names only for Universal class tags, add your
     # own tags using tlv-keyword
     
     #  test this by command:
     #  tlve -c ber.tlverc -s BER <inputfile>
     # or
     #  tlve -c ber.tlverc -s BER -p dump <inputfile>
     
     tl name=asn1ber tag=ber length=ber
     
     structure name=BER content-tl=asn1ber
     tlv name=Boolean tag="U-1" value-type=uint
     tlv name=Integer tag="U-2" value-type=int
     tlv name=BitString tag="U-3" maybe-constructed=yes  value-type=bit-string
     tlv name=OctetString tag="U-4" maybe-constructed=yes value-type=hex
     tlv name=NULL tag="U-5" value-type=hex
     tlv name=ObjectIdentifier tag="U-6" value-type=oid
     tlv name=ObjectDescriptor tag="U-7" value-type=hex
     tlv name=External tag="U-8"
     tlv name=Real tag="U-9" value-type=hex
     tlv name=Enumerated tag="U-10" value-type=int
     tlv name=EmbeddedPvd tag="U-11" value-type=hex
     tlv name=UTF8String tag="U-12" maybe-constructed=yes value-type=string encoding=UTF8
     tlv name=RelativeOid tag="U-13" value-type=hex
     tlv name=Sequence tag="U-16"
     tlv name=Set tag="U-17"
     tlv name=NumericString tag="U-18" value-type=string
     tlv name=PrintableString tag="U-19" value-type=string
     tlv name=T61String tag="U-20" value-type=string
     tlv name=VideotexString tag="U-21" value-type=string
     tlv name=IA5String tag="U-22" value-type=string encoding=ISO646-FI    # change encoding to suit your needs
     tlv name=UTCTime tag="U-23" value-type=string
     tlv name=GeneralizedTime tag="U-24" value-type=string
     tlv name=GraphicString tag="U-25" value-type=string
     tlv name=VisibleString tag="U-26" value-type=string
     tlv name=GeneralString tag="U-27" value-type=string
     tlv name=UniversalString tag="U-28" value-type=string
     tlv name=CharacterString tag="U-29" value-type=string
     tlv name=BMPString tag="U-30" value-type=string encoding=UCS-4
     structure-end
     
     # printing definitions
     # XML produced using this may not be wellformed
     print name=XML file-start="<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>\n" \
           value="<%n>%v</%n>\n" constructor="<%n>\n" constructor-end="</%n>\n" encoding="ISO-8859-1" indent="  "
     
     print name=dump file-start="--- File: %f ---\n\n" value="%n=%v (tag: %d, value %D)\n" constructor="%n (tag: %d)<\n" constructor-end=">\n" indent="  "

This is the first few lines from output of the command ‘tlve -c ber.rc -s BER -p XML /usr/share/beid/certs/beid-cert-government.der’: 

     $tlve -c ber.rc -s BER -p XML  /usr/share/beid/certs/beid-cert-government.der
     <?xml version="1.0" encoding="ISO-8859-1"?>
     <Sequence>
      <Sequence>
       <[C-0]>
        <Integer>2</Integer>
       </[C-0]>
       <Integer>4408622999086524144</Integer>
       <Sequence>
        <ObjectIdentifier>1 2 840 113549 1 1 5</ObjectIdentifier>
        <NULL></NULL>
       </Sequence>
       <Sequence>
        <Set>
         <Sequence>
          <ObjectIdentifier>2 5 4 6</ObjectIdentifier>
          <PrintableString>BE</PrintableString>
         </Sequence>
        </Set>
        <Set>
         <Sequence>
          <ObjectIdentifier>2 5 4 3</ObjectIdentifier>
          <PrintableString>Belgium Root CA</PrintableString>
         </Sequence>
        </Set>
       </Sequence>
       <Sequence>
        <UTCTime>030127000000Z</UTCTime>
        <UTCTime>090127000000Z</UTCTime>
       </Sequence>
       .
       .
       .
In this example an ASN.1 DER coded certificate was converted to XML.

3 Invoking tlve

The format for running the tlve program is: 

     tlve option ... file ...

tlve supports the following options:

--configuration=name
-c name
Read configuration from name instead of ~/.tlverc.
--debug
-d
Dump unprocessable data to tlve.debug.
--name-list=list
-n
Print only elements having name or tag in comma separated list list. tlve recognises the name if it is given with keyword tlv in configuration file. A tag must be enclosed in [ ].
--structure=name
-s
The input data is processed using structure information name. If not given tlve uses structure named as default.
--expression name=value
-e
Expression can be used to select blocks of elements. A block can be defined by by option -l,--start-level and/or -n,--name. If -l or -n is not given, only names appearing in this option are printed. A block is printed if any of given expression evaluates true.

name is name of a primitive element and the expression evaluates true if the contents of the element matches the regular expression in value.

--and
-a
As default multiple expressions are combined with logical or. If this option is given multiple expressions are combined with logical and.
--print=name
-p
Output is formatted using printing definition name. Default is to use printing definition named as default.
--output=name
-o name
Write output to name instead of standard output.
--start-level=level
-l level
Only elements which are in level level or lower in element hierarchy are printed. First (highest) level is level number one.

If -e, --expression is used with this option and the expression(s) in level level or in lower levels evaluates true, then all elements in level or lower levels are printed in that particular hierarchy.

If both this option and -n, --name are defined, only names which appear in level or lower levels are printed.

--stop-level=level
-L level
Only elements which are in level level or higher in element hierarchy are printed. First level is level number one.

If both this option and -n, --name are defined, only names which appear in level level or higher in element hierarchy are printed.

--help
-?
Print an informative help message describing the options and then exit successfully.
--version
-V
Print the version number of tlve on the standard output and then exit successfully.
All remaining options are names of input files, if no input files are specified or '-' is given, then the standard input is read.

4 Configure tlve

tlve uses a text-based configuration file to process and print input data. Default configuration file is ~/.tlverc. Configuration is made up of keywords and parameter-value pairs. Comments start with #-character.

Following chapters explain keywords and their meanings.

tl (tag-length)

tl name=string tag=type,length,mask,shift,offset type=type,length,mask,shift,offset length=type,length,mask,shift,offset print=name value-term=string tl-included=yes|no type-map=name

tl is used to specify characteristics of tag-length pair, which is used to parse a tlv file. Parameters and their meanings:

name=string
Name of this definition, this is used when this definition is referred in other parts of the configuration.
tag=type,length,mask,shift,offset
Characteristics of tag. Typically a tag is defined by its type (integer, printable string,...) and its length.
type
Can be any of following:
int
int-be
Signed integer, big endian.
int-le
Signed integer, little endian.
uint
uint-be
Unsigned integer, big endian.
uint-le
Unsigned integer, little endian.
string
Printable string.
ber
ASN.1 BER encoded tag.

length
The length of the tag as a decimal integer. If the tag and length are separated the separator character can be given in format /c/ where c is the character separating tag and length.
mask
An integer tag is and-masked with this value. Mask can be given as decimal or hexadecimal integer. Hexadecimal values are given as 0xnn. Value zero will not be used.
shift
An integer tag is bit shifted using this value. If shift is positive, tag will be shifted left, if negative tag will be shifted right.
offset
As default the tag is assumed to be in offset 0 of a tlv triplet. if offset is given, tag will be read from position specified by this offset.
Only the type and length or value-term are mandatory, all other values are optional.
type=type,length,mask,shift,offset
If input data contains value encoding information, it can be read using this definition. Input types can be mapped to internal value types using typemap.
type
Can be any of following. Note: Integers are always read as unsigned.
int
int-be
Signed integer, big endian.
int-le
Signed integer, little endian.
uint
Unsigned integer, big endian.
uint-be
uint-le
Unsigned integer, little endian.
string
Printable string.
ber
ASN.1 BER encoded length.

length
The length of the type part of a tlv triplet. If the type and the length are separated, the separator character can be given as /c/ where c is the character separating the type and the length.
mask
The type is and-masked with this value. Mask can be given as decimal or hexadecimal integer. Hexadecimal values are given as 0xnn. Value zero is not used.
shift
The type it is bit shifted using this value. If shift is positive, type will be shifted left, if negative type will be shifted right.
offset
As default the type is assumed to be positioned right after the tag, If this parameter is given type is read using this offset. Note: Offset is relative to the beginning of the tlv triplet, not relative to the end of the tag.
Only the type is mandatory, all other values are optional.
length=type,length,mask,shift,offset
Characteristics of length. Typically a length is defined by its type (integer, printable string,...) and its length.
type
Can be any of following. Note: Integers are always read as unsigned.
int
int-be
Signed integer, big endian.
int-le
Signed integer, little endian.
uint
Unsigned integer, big endian.
uint-be
uint-le
Unsigned integer, little endian.
string
Printable string.
ber
ASN.1 BER encoded length.

length
The length of the length part of a tlv triplet. If the length and the value are separated, the separator character can be given as /c/ where c is the character separating the length and value.
mask
The length is and-masked with this value. Mask can be given as decimal or hexadecimal integer. Hexadecimal values are given as 0xnn. Value zero is not used.
shift
The length it is bit shifted using this value. If shift is positive, length will be shifted left, if negative length will be shifted right.
offset
As default the length is assumed to be positioned right after the tag (or type if defined), If this parameter is given length is read using this offset. Note: Offset is relative to the beginning of the tlv triplet, not relative to the end of the tag.
Only the type is mandatory, all other values are optional.
print=name
Values read using this tag-length definition are printed using printing definition name.
value-term=string
The length will not be used. Instead the value is terminated by string. string is not considered as a part of the value. This parameter and length are mutually exclusive.
tl-included=yes|no
The tag-length part is included/not included in length. Default is value is no, meaning that the length contains only the length of the value part of tlv triplet.
type-map=name
Mapping of input data types to internal types is done using mappings in typemap definition name.

typemap

typemap name=string
map ...
map ...
map ...
.
.
.
typemap-end

name=string
The name of a typemap. This name is used when referring to this item in other parts of configuration
typemap is used to map input value types to tlve's internal value types. Input value type can be defined in tl with keyword type. map keyword has two parameters:

map value=string value-type=int|int-be|int-le|uint|uint-be|uint-le|string|hex|hexs|dec|bcd|bcds|oid|bit-string|escaped
value=string
If string is found as input data type it is mapped to internal value type defined by parameter value-type.
value-type=int|int-be|int-le|uint|uint-be|uint-le|string|hex|hexs|dec|bcd|bcds|oid|bit-string|escaped
This value is used in formatting the value for printing. See the meanings of different value types in next chapter.

tlv (tag-length-value)

tlv name=string tag=string end-tag=string path=string type=constructed|primitive|end-of-content value-type=int|int-be|int-le|uint|uint-be|uint-le|string|hex|hexs|dec|bcd|bcds|oid|bit-string|escaped content-tl=name print=name encoding=name form=definite|indefinite format=string value-length-adjust=integer maybe-constructed=yes|no hold=yes|no|name

tlv is used to specify characteristics of one tlv triplet by mapping additional info to it using the parameter tag as a key. tlv is only used inside the structure definition (see below). Parameters and their meanings:

name=string
The name of a triplet. This name is used when referring to this item in other parts of configuration or in command line (options -n, --name-list and -e, --espression). string is printed using the printing directive %n.
tag=string
string is used as a key when mapping tlv triplet from input to this definition. ASN.1 BER tags are given in form X-n where X is the class of tag: U,A,C or P for Universal, Application, Context-specific or Private type of BER-tag. n is decimal number.
end-tag=string
If given, the string from the parameter tag and string from this parameter are considered as value range when mapping tlv triplets to this definition.
path=string
Only tlv triplets which have path string are mapped to this definition. Path is dot separated list of constructor element names or tags which are preceding this element in constructor hierarchy. If name is not given for a constructor the tag enclosed in [ ] is used. If path starts with asterisk (*) then only the rest of the path must match with the current path.
type=constructed|primitive|end-of-content
The role of this element in tlv hierarchy.
constructed
Element contains one or several other elements, not a single value.
primitive
Element contains a single value, this is the default type.
end-of-content
Elements role is to indicate the end of constructed data. This is normally used when constructor has no given length. These elements are not printed.

value-type=int|int-be|int-le|uint|uint-be|uint-le|string|hex|hexs|dec|bcd|bcds|oid|bit-string|escaped
Type of the value. Default is to use value type from typemap or if that is not used then escaped is used.
int
int-be
Big endian signed integer.
int-le
Little endian signed integer.
uint
uint-be
Big endian unsigned integer.
uint-le
Little endian unsigned integer.
string
Printable string.
hex
Value bytes are printed as hexadecimal numbers.
hexs
Value bytes are printed as hexadecimal numbers with half bytes swapped.
dec
Value bytes are printed as space separated decimal numbers
bcd
Value contains a bcd number.
bcds
Value contains a bcd number with half bytes swapped.
oid
Value contains an ASN.1 object identifier. Sub-identifiers are printed as space separated decimal values.
bit-string
Value bits are printed in groups of eight. For non ASN.1 data all bytes are printed. For ASN.1 bit-string the first byte contains the number of non printable bits of the last byte and it is not printed.
escaped
Value is printed as string, except non printable characters are printed as \xnn, where nn is the hexadecimal value of a character.

content-tl=name
Valid for constructor type elements only. tlv triplets inside this constructor are read using tl definition name. This makes it possible to a single file have different tlv encoding in different levels of constructor hierarchy.
print=name
Printing is done according printing definition name.
encoding=name
The value part of tlv triplet has encoding name. Before printing the value (printing directive %v) the contents of the value is converted to system encoding (default) or encoding specified in printing definition. For more information about encoding see ‘man iconv’.
form=definite|indefinite
Valid for constructor type elements only:
definite
Use constructor elements length to detect the end of constructed content.
indefinite
Constructed element ends when a end-of-content element is found.

format=string
string is a printf-style printing directive for integer values. Integers are handled as long long int and the default formats are %lli and %llu. If string starts with + or ++ the rest of the string is considered as format string for strftime() function and the value is considered to be UNIX time. If single + is given time is formatted as local time, if ++ is given the time is formatted as UTC time.
value-length-adjust=integer
Before reading the value from input, the length of the value is adjusted as length = length + integer. integer can be negative allowing reading less data as expressed by the length in tl data. Note: length consumed by the tlv triplet is not affected by this parameter.
maybe-constructed=yes|no
If an element is a primitive type by default and this is set as yes, then tlve first checks if the beginning of the value is a valid tl pair and the tl pair's length added to the length expressed by the tl pair matches the total length of this element. If this is true the element is treated as a constructed element. Default is no.
hold=yes|no|name
If yes or name the value part of a primitive element or the name of a constructed element is put in hold. The value in hold can be printed using notation $name, where name is the name of the tlv triplet or an alias name given by name. Only the latest value encountered is saved. Several different elements can share the same alias name. This can be used to save the value or the name for later printing. Default is no.
Only parameters name and tag are mandatory.

structure

Structure is a multi-line configuration element, which specifies basic characteristics of a tlv input file. Structure can contain multiple tlv keywords.
structure name=name content-tl=name print=name filler=string hex-caps=yes|no
tlv ...
tlv ...
tlv ...
.
.
.
structure-end

name=name
A name for structure. This is the name which should be given with option -s, --structure.
content-tl=name
A name of a tl definition. tlv triplets are read using this information.
print=name
Elements in this structure are printed using printing definition name, default is to use printing definition named as 'default'. This can be overridden using different names for print in tl or tlv keywords.
filler=string
strings are skipped when reading the input. Non printable characters can be escaped as \xnn, where nn is characters hexadecimal value.
hex-caps=yes|no
if set as yes then the hexadecimal values are printed in capital letters.
After structure keyword there can be any number of tlv keywords, structure definition is ended by structure-end keyword. Only parameters name and content-tl are mandatory.

print

Keyword print specifies how to print tlv triplets in text-based formats.
print name=string file-start=string file-end=string constructor=string constructor-end=string value=string indent=string encoding=name block-start=string block-end=string separator=char

name=string
A name for this printing definition. This name can be referred in other parts of the configuration or with option -p, --print.
file-start=string
Output string for every input file before processing the file.
file-end=string
Output string for every input file after processing the file.
constructor=string
string is printed when a constructor type element is encountered.
constructor-end=string
string is printed after the last element of a constructor type element has been printed.
value=string
string is printed for each primitive tlv triplet
indent=string
string is printed n times before any other data. n is the depth of the constructor hierarchy. Default is not to indent.
encoding=name
If the value to be printed has a encoding parameter in tlv, the name is used as target encoding. Default is to used system encoding (which can be checked by tlve -V).
block-start=string
string is printed before a group of elements found using expression (-e,--expression) and/or start level (-l,--start-level) is printed. Note: string is not printed if group of elements found is empty.
block-end=string
string is printed after a group of elements found using expression (-e,--expression) and/or start level (-l,--start-level) is printed. Note: string is not printed if group of elements found is empty.
separator=char
char is printed after every primitive element except the last one in a group of elements found using expression (-e,--expression) and/or start level (-l,--start-level).
printf-style directives

string used in parameters for print can contain following printf-style directives:

%%
Print single %.
%$
Print single $.
%>
Print the current level of the element hierarchy, first (highest) level is 1.
%l
Print the length of the value part of the tlv triplet.
%c
Print the number of bytes consumed by the tlv triplet.
%t
Print the tag of the tlv triplet.
%n
Print the name (given by the tlv keyword) of the tlv triplet. If the tlv triplet cannot be associated with any of the tlv keywords in structure definition, then the tag enclosed in [ ] is printed.
%v
Print the value part of the tlv triplet. Before printing possible character set conversion is performed.
%p
Current constructor path is printed. Note: This is not printed if option -e, --expression is used.
%o
Print the current input file offset. Offset is the start of the current tlv triplet.
%O
Print the total offset calculated for all input files. Offset is the start of the current tlv triplet.
%f
Print the current file name.
%s
Print the current structure name.
%d
Print the hex dump of the tag-length part in format xnnxnn.... nn is the hexadecimal value of an octet.
%D
Print the hex dump of the value part in format xnnxnn.... nn is the hexadecimal value of an octet.
string can also contain following escape codes: \a, \b, \t, \n, \v, \f, \r, \" and \#. A backslash can be escaped as \\. If string contains white-space characters string must be enclosed in double quotation marks ". Non printable characters can be escaped as \xnn, where nn is characters hexadecimal value.
Hold variables

If elements is put in "hold" (parameter hold in tlv-keyword), then the value of a primitive element or the name of a constructed element can be printed in strings using notation

$name

where the name is the name of the tlv triplet or the alias name. Tlve prints always the latest encountered value or name of the name. This can be used e.g. to save a value from the beginning of the file to be printed with later found elements.

Input Preprocessor

It is possible to define an input preprosessor for tlve. An input preprocessor is simply an executable program which writes the contents of the input file to standard output which will be read by tlve. If the input preprosessor does not write any characters on its standard output, then tlve uses the original file.

To set up an input preprocessor, set the TLVEOPEN environment variable to a command line which will invoke your input preprocessor. This command line should include one occurrence of the string %s, which will be replaced by the input filename when the input preprocessor command is invoked.

The input preprocessor is not used if tlve is reading standard input.

Convenient way is to use lesspipe (or lesspipe.sh), which is availabe in many UNIX-systems, for example 

     export TLVEOPEN="/usr/bin/lesspipe %s"

Using the example above is it possible to give a zipped input file to tlve, then the input processor will unzip the file before it is processed by tlve.

5 How to use tlve

Examples of rc-files can be found in tlve package.

6 Reporting Bugs

If you find a bug in tlve, please send electronic mail to tjsa@iki.fi. Include the version number, which you can find by running ‘tlve --version’. Also include in your message the output that the program produced and the output you expected.

If you have other questions, comments or suggestions about tlve, contact the author via electronic mail to tjsa@iki.fi. The author will try to help you out, although he may not have time to fix your problems.

Table of Contents