It's remained seared in my consciousness since my first visit in 2004.
I wasn't the only one impressed; the city became a UNESCO World Heritage Site in 2006.
A decade later, following much development and change in Ethiopia, I was curious to see if the city still retained its exotic finesse and character that so affected me before.
So I returned.
From my hotel room's small balcony I could see the Asmaddin Beri (beri means gate, as well as, rather more grandly, portal).
It's one of six punctuating the thick five-meter-high walls running 3.5 kilometers around the Jugal, the name for the 16th-century fortification that lies within the modern town that developed from the 19th century onward.
Once through the Asmaddin portal the 21st century vanished, replaced by a sense of antiquity and a heaving, shambolic outdoor market, one of many dotted around the Jugal.
Harari women in colorful dresses squatted beside neat piles of onions, tomatoes, green peppers, bananas and more.
Sweet smells wafted from where women sold pots of itan (incense), while samosas cooking on small stoves and baskets full of fresh bread rolls added to the pleasant stimulation.
"Feranju! Feranju! Amantekhi?!" the women called in the local Harari dialect, which roughly translates as: "Foreign guy, foreign guy, how are things?!"
The simplest way to explore the Jugal is to get lost among the warren of narrow, twisting alleys.
Eventually you stumble upon a main street or landmark.
On Mekina Girgir, a narrow, atmospheric street packed with tailors' workshops and men at sewing machines, one tailor looked up from his machine to point the way to what's known as Arthur Rimbaud's house.
The French poet, who turned his back on literary stardom and became a gunrunner in Harar, didn't actually live in the large ornate house standing today.
This was built by an Indian merchant on the site of an earlier house where Rimbaud apparently lived.
Inside, on the second floor, a photographic exhibition of turn-of-the-20th-century Harar shows the strange world Rimbaud encountered -- some scenes with similarities to the present day.
Despite the Jugal containing 82 mosques and this being a return trip I couldn't seem to find any of them as I zigzagged among pastel-colored alleyways that decorate the walled city's interior.
I enlisted the help of a young scruffy boy passing by who, within 10 minutes, had taken me to three mosques.
At each one he beckoned to where he indicated the best position from which to take a photograph.
He was right every time.
Center of Islamic scholarship
Believed to have been founded by Arabian immigrants around the 10th century, Harar became a crossroads for trade and culture, ideally situated between the Ethiopian highlands to the west and the shores of the Gulf of Aden to the east.
The city evolved into a center of Islamic scholarship and culture, and eventually was considered a sort of capital city of Islamic northeast Africa.
In the 16th-century, Emir Nur ibn al-wazir Mujahid became Harar's ruler and fortified the city against the threats of Christian forces from Ethiopia and increasing migration of the Oromo people.
Foreigners -- especially infidels -- were denied access to the city.
But in 1855 British explorer Richard Burton ignored the ban and traveled from the port of Zeila on the Somalian coast to enter the city disguised as an Arab merchant through Argob Beri in the eastern wall, staying for 10 days.
Nowadays locals are more accepting of those coming from far beyond their walls.
After a decade of rising tourist numbers, I noticed how more people, children and adults included, expected to be paid for having their photos taken.
I exhausted my quota of polite rebuttals to the stream of offers from young men offering to act as guides.
Despite some residents changing their attitudes toward tourists, the general way of life in the Jugal seemed utterly unchanged from my 2004 visit.
I still seemed a long way from the international commerce and high-rise buildings that have emerged in Addis Ababa, Ethiopia's rapidly growing capital.
Oromo women still came from the surrounding rural areas bringing donkeys laden with firewood to sell and then spend their earnings in the Jugal's markets on food and household goods.
Sharp-eyed kites still circled around the Gidir Magala market and the city's main butchery, ready to swoop down on any stray meaty tidbits.
Not everyone expected money for a photo.
Twelve-year-old Enas Fysel wouldn't stop posing for photos after finishing her game of marbles in an alley, before insisting on taking me on a tour of her neighborhood.
Once outside the Jugal and back in the modern city I slipped back into my 21st-century ways, choosing a particular restaurant for dinner because I knew it had free Wi-Fi.
When I first visited Harar I didn't carry a laptop in my backpack.
My camera contained film.
It's not just places we visit that change, evidently. So do travelers, and how we travel.
How to get to Harar:
By plane: Ethiopian Airlines flies to Dire Dawa city (one-hour flight; $300 for a return). Shared minibuses ($1-$2) running frequently through the day connecting to Harar (takes about one hour). Alternatively taxis can be hired ($15) from Dire Dawa to reach Harar more quickly.
By Bus: Modern long-distance coaches operated by Sky Bus or Selam Bus (nine to 10 hours, $25 for a one-way ticket), depart 5:30 a.m. from their ticket offices in Addis Ababa at Meskal Square). Smaller, older buses depart around 6 a.m. from the main bus station in the Merkato market. Though cheaper ($9), they're not as comfortable a ride -- especially for taller passengers. All buses terminate at the bus station in Harar next to Harar Gate. The minibuses that do the Addis Ababa-Harar run are quicker and can be organized to pick up passengers from their hotel -- for a commission.
James Jeffrey is a freelance journalist based in Addis Ababa from where he writes about Ethiopia and the Horn of Africa for various international media.