There are places we visit long before we ever stop there.

As a child, every summer we drove from Cambridge to Holkham Beach on the north Norfolk coast in my father’s silver Citroën DS Safari estate. It was a magical machine, appearing to float rather than drive, its famous hydropneumatic suspension ironing away every imperfection in the road. My brother Jared and I sat in the back watching East Anglia unfold through the enormous windows as villages drifted past like illustrations from a favourite book.

One of those villages was Lavenham.

Even then it seemed to belong to another century. I remember the astonishing jumble of black-and-white timber-framed houses leaning improbably towards one another over narrow streets, as though centuries of conversation had caused them to bend together. Somewhere among them stood The Swan, although I didn’t know its name then. We slowed as we drove through the village before continuing north towards the sea. My father, who loved architecture almost as much as he loved driving, would often remark upon the extraordinary buildings. Yet somehow we never stopped. Lavenham became part of my childhood geography without ever becoming part of my experience.

It has taken me more than fifty years to put that right.

The reason for returning was a happy one. My mother, the sculptor Helaine Blumenfeld, had just opened a major exhibition at Gainsborough’s House in nearby Sudbury. Her latest sculptures, carved from luminous white Statuario marble during the two years since my father’s death, are meditations on grief, resilience and renewal. Seeing them installed in the house where Thomas Gainsborough was born felt unexpectedly appropriate. Both artists, in their very different ways, understand how light can transform matter into emotion.

Sudbury itself is a handsome Suffolk market town. Were it not for Lavenham sitting only a few miles away, it would undoubtedly be remembered as one of the loveliest places in East Anglia. But comparisons are unavoidable.

The village owes its existence to sheep. During the fifteenth and early sixteenth centuries it became one of the richest settlements in England through the wool trade, exporting the famous cloth known as Lavenham Blue across Europe. Its wealthy cloth merchants built magnificent timber-framed houses, an imposing Guildhall and the soaring Church of St Peter and St Paul, whose tower still dominates the surrounding countryside. Then the trade collapsed. Fashion changed, competition increased and the money disappeared. Had prosperity continued, much of mediaeval Lavenham would almost certainly have been demolished and replaced by Georgian elegance or Victorian confidence.

Instead, the town simply stopped.

Ironically, poverty became its salvation. Without money to modernise, Lavenham remained almost exactly as it had been. More than three hundred listed buildings survive today because nobody could afford to replace them.

Walking through the village is like stepping inside a mediaeval painting. The streets refuse to run straight. The houses lean at impossible angles. Rooflines dip and rise. Windows appear to have settled wherever their builders thought they looked best. Nothing is quite level, yet everything feels perfectly balanced.

At the centre of it all stands The Swan.

Part fifteenth-century merchant’s house, part coaching inn, part architectural maze, it has welcomed travellers for hundreds of years. To describe it simply as a hotel rather misses the point. Staying here feels less like checking in than being admitted, briefly, into six centuries of English life.

The building immediately asks you to abandon modern expectations. Ceilings are low enough to demand an instinctive bow. Oak beams, blackened with age, stretch overhead with immense confidence. Corridors meander. Staircases appear unexpectedly before vanishing around blind corners. One passage climbs three steps only to descend four. Every generation has added another room, another wall, another doorway until the whole building feels less designed than accumulated.

It is gracefully higgledy-piggledy. It is also, inevitably, difficult. Accessibility is limited, although it is hard to imagine how it could be otherwise. Even bedrooms described as being on the ground floor involve unexpected steps and changes in level. Mediaeval carpenters were not anticipating wheelchairs, lifts or rolling suitcases. They were simply building something that would outlive them.

During the exceptional July heatwave, my mother’s room became almost unbearably warm.

“Yes,” Leanne on reception admitted cheerfully, “it’s boiling in these mediaeval houses during summer. And freezing in winter.”

There was something refreshing about the honesty.

Our own room was in what the hotel described as the newer part of the property.

Only in Suffolk could “new” mean four hundred years old.

The bedroom had been created directly beneath the roof, enclosed within an extraordinary forest of ancient oak beams. They rose above us like the inverted hull of a mediaeval ship, each one hand-hewn, each one carrying the marks of craftsmen who could never have imagined that, centuries later, strangers would lie beneath them photographing them with mobile phones.

Because the room sat immediately below the roof tiles, it was also the only bedroom in the hotel fitted with air conditioning. It was very welcome. Even so, the great timbers seemed to radiate centuries of stored warmth.

If I have ever stayed in a room that might genuinely have been haunted, this was probably it.

Nothing supernatural occurred. No ghostly footsteps echoed along the corridor. No mysterious figures appeared at the foot of the bed. Yet buildings of this age possess an undeniable presence. As darkness settled over Lavenham, the oak beams creaked softly as temperatures changed, sounding exactly as they must have sounded to merchants, pilgrims, coachmen and travellers long before any of us arrived.

Sleep, however, proved more elusive than the ghosts.

The mattress had developed an alarming tendency to collapse towards its centre while simultaneously dipping at the edges, making comfortable sleep almost impossible. It was, without question, the least successful part of our stay and memorable for entirely the wrong reasons.

Then, lying awake beneath those magnificent sixteenth-century beams, another question occurred to me.

How on earth had anyone ever managed to get a king-size mattress into the room?

Looking back towards the impossibly low doorway, through the twisting passages and narrow staircases, the logistics seemed almost as mysterious as any haunting. Some puzzles are perhaps better left unanswered.

Dinner that evening reminded us why The Swan enjoys such an excellent culinary reputation. Served beneath soaring timber beams in one of England’s most atmospheric dining rooms, the meal struck exactly the right balance between refinement and restraint. I began with a delicious coconut and squash soup before moving on to beautifully prepared fresh sea trout, expertly filleted from the bone. It was cooking confidently enough not to show off, allowing excellent ingredients to remain the principal attraction.

Breakfast the following morning, served beneath the same magnificent hammer-beam ceiling, felt almost ecclesiastical. Morning sunlight streamed through leaded windows, illuminating details in the ancient oak framework that had disappeared into shadow the previous evening. Few hotel breakfast rooms possess quite so much history before the coffee arrives.

Afterwards we wandered into the Market Square to visit the magnificent Grade I-listed Guildhall, now cared for by the National Trust. Built at the height of Lavenham’s prosperity by the Guild of Corpus Christi, it remains one of England’s finest surviving mediaeval guildhalls. Like the village itself, its survival owes as much to economic decline as careful preservation. It stands today not because history was kind but because history ran out of money.

A short walk away rises the equally magnificent Church of St Peter and St Paul. Its scale still surprises. This is not the church one expects to find serving a village. Rather, it is the church of a place that once believed itself among the wealthiest communities in England and wanted heaven to know it.

That afternoon we returned to Sudbury for my mother’s exhibition. Seeing her sculptures after walking through Lavenham created an unexpected conversation between two very different kinds of permanence. One is carved from marble. The other is from oak. Both have survived because their makers believed beauty deserved time.

For our second evening we drove a few miles to The Peacock at Chelsworth, another ancient Suffolk inn, though one with an altogether different personality. Where The Swan envelops you in mediaeval grandeur, The Peacock offers quieter elegance.

Its restaurant proved exceptional.

Several friends joined us, choosing everything from pea risotto to kingfish fillet, and by the end of the evening more than one independently declared it among the finest meals they could remember. I began with an exquisite salad of melon and pickled cucumber, beautifully judged in the summer heat, before another perfectly cooked kingfish. Dessert was a sublime dark chocolate mousse. One member of our party thought it insufficiently sweet. I thought it exactly right, allowing the quality of the chocolate to speak for itself rather than burying it beneath sugar.

Driving home, I found myself thinking once again about those journeys north with my father in the Citroën. For decades Lavenham had existed only as a fleeting glimpse through a car window. I could never have imagined that I would one day return with my mother, celebrating an exhibition shaped by our family’s own experience of love and loss, before sleeping beneath beams that had quietly watched England change for six hundred years.

The mattress, admittedly, may have been the youngest thing in the room, but it had aged considerably less successfully than the fifteenth-century oak above it.

Fortunately, mattresses can be replaced.

Six centuries of history cannot.