ISO8583 get started, all you need to know

Introduction

I remember back in 2017, I first heard about ISO 8583 in the context of Mastercard issuing and processing. At the time, I knew absolutely nothing about the financial world and the thought of tackling something like ISO 8583 seemed pretty scary. But, as I started researching and learning about the standard, I quickly realized that it’s actually not as complicated as I initially thought. Sure, there are some complexities when it comes to exceptions and subfields, but overall, I found the standard to be pretty simple and straightforward.

ISO 8583 is a widely-used international standard for financial transaction messaging. It defines a common format for the exchange of financial transaction information between different systems and organizations. In this blog post, we will explore the key features of ISO 8583 and its applications in the world of card payment systems.

Over the past five years, I’ve had the opportunity to work with a variety of libraries and tools that implement or support the ISO 8583 standard. I’ve learned a lot during that time, and in this blog post, I also want to give back to the community by sharing some of the examples I’ve picked up along the way. It’s my hope that the examples I share will help others who are also working with ISO 8583, and make the process of understanding and implementing the standard a little bit easier.

ISO8583 in the nutshell

ISO 8583 defines a standard format for financial transaction messages, including information such as the account number, transaction amount, and approval code. This standard format allows different systems and organizations to communicate and exchange financial transaction information in a consistent and reliable way.

One of the main advantages of ISO 8583 is its flexibility. The standard includes a number of fields that can be used to convey specific information about a transaction, such as the type of card used or the merchant’s location. This allows organizations to use the standard to support a wide range of different types of transactions and business requirements.

Imagine you’re at a store and you want to buy something. You hand over your card to the cashier and they swipe it through their POS terminal. The information from your card, like your card number and your personal details, along with information from the terminal and the transaction details, like the amount, gets sent through a series of networks to the card issuing system. This system then checks if you have enough funds in your account to cover the purchase and either approves or declines the transaction. Once the decision is made, a response message is sent back to the POS terminal, letting the cashier know whether or not the transaction was approved. After you got your product or service and your transaction is approved that is not the fact that money actually moved from your issuing bank to acquainting bank it happens later on during settlement procedure.

This all lifecycle handled throw ISO 8583 standard.

ISO 8583 message is consist of 3 main parts.

• MTI (message type indicator)
• Bitmap(s) (indicating which data element are present in the message, sometime message has 2 or 3 bitmaps primary and secondary, the first bit of the primary bit map shows whether secondary bitmap present or not)
• Data Elements (actual data fields which contain information about the transaction.

MTI

It is a 4 digit filed consists of numbers, which indicates the meaning/type of the specific message.

• 1st digit (Version)

Indicates which version of the standard the message is encoded. (there are other meanings for 1st digit of MTI however in this post we wont talk about them, check here to know more about different possible values)

0*** - ISO 8583:1987
1*** - ISO 8583:1993
2*** - ISO 8583:2003
...

• 2nd digit (Message class)

The next digit in the MTI shows general purpose/type of the message.

*0** - Reserved by the standard
*1** - Authorization
*2** - Financial
*3** - File actions
*4** - Reversal and chargeback
*5** - Reconciliation
*6** - Administrative
*7** - Fee collection
*8** - Network management
*9** - Reserved by the standard

For more details check here.

• 3rd digit (Message function) The Message Function is a code that tells what type of action or function the message is trying to perform, such as requesting authorization or sending a financial request. It helps the recipient understand what the message is trying to accomplish.

**0* - Request
**1* - Request response
**2* - Advice 
**3* - Advice response
**4* - Notification
**5* - Notification acknowledgment
**6* - Instruction
**7* - Instruction acknowledgment
**8* - Reserved for standard
**9* - Reserved for standard

What is important to note here is that the response/acknowledgment is always iterated by one from request. E.g. if you get **0* the response should be 0 + 1 **1*, similarly when you get **2* the response should be 2+1 **3* .

See more examples here.

• 4th digit (Originator) 4th digit tells where the message is coming from in the payment process, such as if it’s from the card issuer or the card acquirer.

***0 - Acquirer
***1 - Acquirer repeat
***2 - Issuer
***3 - Issuer repeat
***4 - Other
...

There are more originator values that have been reserved for IOS use only you can find more about them here.

Bitmap(s)

A bitmap is a binary code in ISO8583 that shows which data fields are included in a financial transaction message. Each bit represents a single field, with 1 meaning it’s included and 0 meaning it’s not.

First bit of bitmap indicates whether it consist secondary bitmap or not. The primary bitmap includes information of 1-64 data elements after that secondary bitmap holds from 65-128 data elements in case there are more elements in the message the third bitmap will exist which will indicate the existence of data elements from 129 to 192.

Now about slightly confusing part. In general bitmap can be represented in 8 byte (packed) or 16 byte of hexadecimal (unpacked) format. The difference is really comes to readability of data, however I have seen that sometimes some processors can already send unpacked data in that case you have to configure your packager or library to know how to parse the rest of the message.

Example

Lets say we got following fields in our ISO message: 👇 3, 4, 6, 7, 10, 12, 13, 14, 21, 24, 25, 27, 29, 30, 33, 34, 36, 39, 44, 48, 52, 55, 57, 59, 62,63

if we represent this as binary it will look like this: 👇 64bit = 8byte of binary (packed) bitmap 0011 0110 0101 1100 0000 1001 1010 1100 1101 0010 0001 0001 0001 0010 1010 0110

Then when we make something more human readable and compact we can represent following binary as 16 byte of hexadecimal (unpacked) value which will look like this: 👇 and obviously will take more space. 365C09ACD21112A6

Here you can see how we get hexadecimal form binary.

0011 -> 3
0110 -> 6
0101 -> 5
1100 -> C
... etc

Here you can exercise a bit how the bitmap will look like when some data elements exist or not.

Data Elements

In this article, I’ll give a general overview on how to understand and convert raw data into something readable, without delving too much into specific data elements.

Each data element is specified in a standard format that specifies the allowed data type (such as numeric or alphabetic) and field length (whether it is dynamic or fixed).

When the data element is dynamic length that would mean that first 1, 2 or 3 digits of the data element shows the length of that specific data element.

Commonly in the documentation or in schema’s manuals the data elements are annotated with length indicator L, LL, LLL which folles the type of the data element VAR, INT, … *

For example here 👇 we see J8583 (ISO 8583 library) packager, where we can observe several filed annotations.

1. num="3" type="NUMERIC" length="6" this would mean that data element 3 is numeric with 6 fixed length value.
2. num="32" type="LLVAR" similarly this annotation tells parser that data element 32 has dynamic length and can be from 0-99 length identified by the first 2 digits of data element. So for this specific example the raw filed would look like 03456 where 03 is the length of the message followed by message itself.

<template type="0200">
    <field num="3" type="NUMERIC" length="6">650000</field>
    <field num="32" type="LLVAR">456</field>
    <field num="35" type="LLVAR">4591700012340000=</field>
    <field num="43" type="ALPHA" length="40">Fixed-width data</field>
    <field num="48" type="LLLVAR">Life, the Universe, and Everything|42</field>
    <field num="49" type="ALPHA" length="3">840</field>
    <field num="60" type="LLLVAR">B456PRO1+000</field>
    <field num="61" type="LLLVAR">This field can have a value up to 999 characters long.</field>
    <field num="100" type="LLVAR">999</field>
    <field num="102" type="LLVAR">ABCD</field>
</template>

The example is taken from here

Useful libraries and tools

Enough is enough lets talk about practical stuff!

In this post we’re gonna check out some cool tools and libraries that developers can use to create ISO8583 processing systems like acquiring or issuing. Though I won’t go too deep into each one, but we’ll give you some handy references to check out.

Golang libraries

I haven’t used Golang in any production work yet, but I’ve been curious about the libraries and references under this title for quite some time, and I’m excited to share them with the community. Overall, I believe that Golang has the potential to be the best choice for this purpose due to its simplicity, high performance, and built-in network tools.

Thanks to MOOV we have following ISO8583 library that is very attractive in my opinion, moreover it is actively under development and already proven to be effective in many production cases.

See more here 👇

• MOOV 8583 GitHub link

Simple example of Packing message with MOOV ISO8583

// spec is already predefined specification of ISO messages 
message := NewMessage(spec)
message.MTI("0100")

// set all message fields you need as strings

message.Field(2, "4242424242424242")
message.Field(3, "123456")
message.Field(4, "100")

// generate binary representation of the message into rawMessage and handle errors
rawMessage, err := message.Pack()

✅ One very useful thing about this library is that you can use it in CLI check here. It can be very useful if you are constantly in development and want some lightweight program to marshal/unmarshal messages.

Last but not least among many cool features this library supports encoding messages to JSON, can be very handy in modern development.

JPOS and JPOS EE

JPOS is open source and widely-used Java library for ISO8583 standard-based issuing, processing, and acquiring, with a long-standing presence in the industry.

JPOS has a big benefit in its modular design. It is made with plug-ins, so developers can easily add or remove features as required. This makes it possible to customize JPOS to meet specific needs and add new features with custom plug-ins.

JPOS can be configured at runtime, meaning that its behavior can be modified without having to recompile the code. This is possible because JPOS uses an XML-based configuration file, which can be edited and reloaded while the application is running.

In my opinion, if you are working with stateless microservices, JPOS’s runtime configuration may not be very useful and could actually add complexity. Additionally, integrating JPOS with modern frameworks like Spring or Spring Boot while still taking advantage of its features can be challenging, especially if you want to replace its built-in server. Although JPOS offers an Enterprise Edition, it can be expensive, and it will limit your ability to use all the features of Spring Boot.

Besides that all if you still decide or need to use JPOS there is Github project that would allow you to run JPOS from within Spring Boot. (check it below 👇). Just for a record I have successfully used this library in production however with some tricks and hacks here and there that.

• GitHub to use JPOS with Spring Boot

Javascript library

GitHub link here

Although I don’t have much experience with this library and only used it once out of curiosity, I would like to mention an interesting feature that caught my attention. The library is capable of converting XML ISO messages into JSON objects, which can be very beneficial when integrating with external parties that use XML communication.

ISO8583 simulators

1. Here you can find simulator from JPOS itself.
2. I have used this for quite some time it is quite simple but unfortunately not maintained and a bit buggy. Possible to simulate both client and server. TODO testing tools would be good
3. Neapay also provides quite many useful tool that work with java 8-14.

In my opinion, relying solely on a simulator for ISO8583 messaging may not be effective as simulators may not cover all scenarios, including various exceptions and message types. Additionally, acquirers or issuers may send non-compliant messages that deviate from the specification, making real-life testing different from testing with a simulator.

Jreactive 8583

Finally I really want to talk about Jreactive8583 so far the best Java based solution to reliably use it in production with Spring Boot. The motivation of the author Konstantin Pavlov explains it all: 👇

1. jPOS library is not free for commercial use.
2. j8583 is free but does not offer network client

Jreactive8583 is an ISO8583 client and server that utilizes the power of Netty asynchronous messaging framework and j8583 for parsing messages in both direction.

• It is possible to configure automatically respond sign on messages
• It’s possible to automatically reconnect to the server when the connection is lost for some reason
• Logging is much easier and secure (you can request logging with masked pans also optionally with fields descriptions).
• Interface to work with native natty pipeline

To properly handle message parsing, it is crucial to adjust your message factory settings to fit your processor.

Here an example of message factory configuration in Spring.

@Configuration
public class MessageFactoryConfig {

  @Bean
  public MessageFactory<IsoMessage> messageFactory(){
    //you must specify the standard version and message origin (e.g. ACQUIRER, ISSUER or OTHER.)
    final MessageFactory<IsoMessage> messageFactory = new J8583MessageFactory<>(ISO8583Version.V1987, MessageOrigin.ISSUER);
    //Here we set the character encoding to use for parsing data elements
    messageFactory.setCharacterEncoding(StandardCharsets.US_ASCII.name());
    //Mesages would come as binary
    messageFactory.setUseBinaryMessages(false);
    //We do not want dynamically generate date for each message
    messageFactory.setAssignDate(false);
    //Our bitmap would be binary (packed)
    messageFactory.setUseBinaryBitmap(true);
    //We tell how we will generate Data element 11 STAN (system trace audit number)
    messageFactory.setTraceNumberGenerator(new SimpleTraceGenerator((int) (System.currentTimeMillis() % 1000000)));

    return messageFactory;
  }
}

Now we can configure client with our message factory.

@Configuration
public class DonkeyConfiguration {

  @Autowired
  private final MessageFactory<IsoMessage> clientMessageFactory;

  @Bean
  public void configure() {

    SocketAddress socketAddress = new InetSocketAddress(host, port);
    ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
        //Here we specify that each message will start 
        //with 4 digits showing the length of the message
        .frameLengthFieldLength(4)
        //And the lehgth would be encoded as plain text
        .encodeFrameLengthAsString(true)
        //Will configure client to print ISO fields in the logs
        .describeFieldsInLog()
        //We will give interval to reconnect in case something goes wrong
        .reconnectInterval(reconnectionInterval)
        //When we have an exception we will not rply back
        .replyOnError(false)
        //We tell our client to not log any sensitive data like PAN
        .logSensitiveData(false)
        .build();

    Iso8583Client<IsoMessage> client = new Iso8583Client<IsoMessage>(socketAddress, clientConfiguration,
        clientMessageFactory);

    client.addMessageListener(new IsoMessageListener<IsoMessage>() {
      @Override
      public boolean applies(IsoMessage isoMessage) {
        return true;
      }

      @Override
      public boolean onMessage(ChannelHandlerContext ctx, IsoMessage isoMessage) {
        //Do manipulations with your message such as
        //Check balance limits 
        //Write transactions
        //Book balance
        //Approve the iso message
        //etc..
        ctx.writeAndFlush(responseMessage);
        return false; // should be true if message should go to other handler
      }
    });

    
    client.init();
    client.connect();
    if(client.isConnected()){
      // now we are connected to the server we can send messages using `send` or `sendAsync`
      client.send(isoMessageUtil.generateSignOnMessage(clientMessageFactory));
    }

    client.shutdown();
  }

}

The server is very similar to the client just you do not specify the host. Here the exapmle form documentaiton.

Iso8583Server<IsoMessage> server = new Iso8583Server<>(port, messageFactory);

server.addMessageListener(new IsoMessageListener<IsoMessage>() {
    ...
});
server.getConfiguration().replyOnError(true);
server.init();
server.start();
if (server.isStarted()) {
    ...
}

server.shutdown();

Conclusion

In summary, ISO 8583 provides a standardized format for financial transaction messages, which enables different systems and organizations to communicate and exchange information consistently and reliably. This flexibility allows the standard to support a variety of transactions and business requirements. There are several major tools/libraries we tackled in this blog post which I hope will enable readers to be less confused when they start working with related techonlogies. In the end I want to give kudos to Konstantin Pavlov for such a great library (Jreactive8583) and contribution to the community.