RecordFlow: When File Formats Became a Reusable Framework

I came across another old presentation recently.

This one was about something we had built inside a large Enterprise Data Exchange Platform. Lots of structured files, record definitions, processing rules — not exactly something you would normally dig out years later for fun.

But going through it again reminded me why I had got quite interested in the problem at the time.

The problem was all the repetition

The platform had to deal with a very large family of structured files — getting close to a hundred different record definitions across the wider environment.

Different fields. Different order. Text, numbers, dates. Mandatory and optional fields. Different lengths and formats.

The normal approach was basically to write another program.

Parse this format. Validate those fields. Construct another record. Repeat.

And there were already programs doing exactly that.

They worked.

But after looking at enough of them, what started bothering me was not any one program. It was the pattern across all of them.

A lot of the code was doing essentially the same thing.

What was really changing was the definition of the record.

What if the format itself was data?

That became the idea behind what I will call RecordFlow here.

Instead of putting every format definition inside a program, put the definition itself into data.

What fields are there? In what order? Which are mandatory? What type? How long? What format?

Then build one common framework that understands those definitions.

Give it values, it can build a record.

Give it a record, it can break it apart.

And the same definition can be used to validate what came in.

The thought was actually pretty simple:

instead of teaching every program how to understand its file, teach one framework how to understand a file definition.

Once that clicked, a lot of the repeated mechanics could disappear from the individual programs.

Then it had to work in the real world

This was not small-volume processing either. It sat underneath very large recurring data flows.

So getting the abstraction right was only part of it.

If something stopped halfway through, what had completed? What failed? Why? Can we restart from there? Can we rerun only what needs rerunning? Can independent work happen in parallel?

Audit, errors, restart, rerun, recovery and parallel processing became part of the surrounding design.

Not really a different idea.

More like: once the processing model became common, the operational behaviour could become common too.

I still remember during go-live how pleased the person responsible for the system was with this part.

Not only that it solved the processing problem, but that when something went wrong, the support team could actually understand what happened and recover from it.

It was not an obvious change to make

There were already established programs doing this work.

And a few people quite reasonably asked why we should change something that was already running.

That made this one harder.

My view was not that the old programs were bad. They were not.

It was that every new requirement was creating another version of essentially the same machinery.

Eventually we went ahead with the common approach.

And over time it spread well beyond the original need. Similar processing moved onto the framework, older point programs could be retired, and the approach started getting used for a much broader set of structured data-processing needs.

I also trained a few people on the design, and gradually they started extending it and taking care of it themselves.

What I see in it now

Looking back, I can attach all sorts of current terminology to it — metadata-driven design, declarative processing, configuration over code.

I do not remember thinking about it in those terms.

It really started with observing the pattern properly.

Across all those different formats, most of the machinery was actually the same.

What changed was the definition.

Once I saw that, the next thought was almost natural:

why keep putting the differences into programs when the differences themselves can become data?

That led to the metadata model.

Then came the common programming model around it.

And once that started getting used more widely, the thinking naturally moved beyond the program itself — recovery, audit, parallelism, supportability, reuse, and eventually other people being able to own it.

First see the pattern. Separate what changes from what stays the same. Turn the variation into metadata. Build the common mechanism around it. Then think about the whole system that has to live around that mechanism.

Looking at those old slides now, a lot of the systems thinking I use today feels strangely familiar.

The thing I remember most is not the file processing itself. It is that the design started with seeing the pattern clearly, moved into metadata and a common programming model, and then grew into something other people could operate, extend and eventually own.