They had four children (George, Helena, Edwin, and Walter) and moved to Andover, and in common with Dugay tradition Henry ran a pub called the Wheatsheaf in East Street from 1861 to 1869. He became bankrupt in 1867, but continued working as a carpenter and beer seller until 1869. 

And now things start to go wrong. In March 1883 Henry's wife Sarah died from bronchitis. Henry started up a relationship with another woman, Annie Martin, and transferred the Olive Branch licence to his son George who was a soldier in the Royal Engineers. 

So the writing is on the wall. The licensing magistrates wouldn't allow the father Henry to have a licence, and the two children are unsuitable for different reasons.  Then a few days later: (this is surely not just a coincidence)!

By the end of the year the inevitable happens and the remaining family have to leave the Olive Branch:

Hampshire Advertiser 1st December 1883

Eighteen months later:

Hampshire Advertiser 4th March 1885

This one is difficult to read so here is as transcription:

THE STRUGGLE FOR EXISTENCE - An inquest was held yesterday (Tuesday) afternoon at the Black Swan, licensed house, St. Mary street, touching the death of Florence Maud Dugay, an infant, nearly three months old, whose mother lives in Craven-street. Annie Dugay, wife of Henry Dugay, a carpenter, said her husband had deserted her since the 24th of May last. The child was very small from birth, and did not to seem to have improved. The deceased had fed well and for the last two days she had had a slight hacking cough. Between 6 o'clock and half past that morning witness got up to see the time and laid the child on the pillow of her arm. When she went to take hold of the infant again, almost directly, witness found that the child's legs were drawn up. She went to her next door neighbours and said that she believed her baby was dead, and a little girl was sent for Dr Archer. She returned with a message that the doctor had been up part of the night and she must get someone else. She went across to the Police station. Witness took in washing for her livelihood, earning on an average 2s 6d per week, and she had a son working at Woolston for 6s a week, and another son sold salt and hearthstone. Her oldest daughter allowed her £1 a month, which paid the rent. Her husband was her second and left her because she would not turn her boys out. The deceased had not been kept short of food. She had no support from the parish minister or the Guardians, and when she applied to the relieving office for relief she was told to go into the Work house, but she would not break up her home on account of her boys. She had seven dependent on her, and her son, who sold salt, was the principal bread winner. Annie Fisher, a widow living next door to the mother of the deceased, said she fed the child at half past 10 the previous night, and the child seemed all right. She fed the infant with soaked biscuit. About half past six Mrs Dugay came and told witness that her child was dead. One leg and one arm were contracted. Mrs Dugay had always well fed the child and was a kind mother. Before the child was born Mrs. Dugay asked for medical attendance, but it was refused because her husband was away. The husband left on the 23rd May, and the child was born the 17th December.  By the jury : Mrs. Dugay was only married to the second husband about two months before he went away, and she was widow about four years. The jury could not come to any conclusion, and the inquiry was adjourned till Thursday afternoon to enable for a post mortem examination to be made.

Portsmouth Evening News 6th March 1885

Then a couple of years later - another  pub - also called the Olive Branch!

Reading Mercury 26th August 1899

Recently returned from America? Maybe he went back into the Merchant Navy but we haven't found any trace of him in this missing period.

Reading Mercury 2nd September 1899

Henry's tenure of the second Olive Branch lasted only for a few months:

Reading Mercury 27th January 1900