Social worker Pamela Campbell said the two women also engaged in self-recrimination for not having kept a closer eye on Millie.
Millie's mother, 27-year-old Rachael Martin, from Main Street, Kesh, and her former partner, 33-year-old Barry McCarney, from Woodview, Trillick, deny charges linking them to Millie's death in December 2009.
Mrs. Campbell said that as she was talking to the distraught Martin and her mother, Margaret Graham, "the penny seemed to be dropping" and they began "looking back over the past few weeks".
The social worker said both mother and daughter "became angry ... calling him (McCarney) names" while at another stage "they swore about him".
She said "they were both extremely distressed and very angry ...that someone could have done this to Millie ...they were extremely distraught and going over things".
Mrs. Campbell said they complained that recently, when they went to pick Millie up, she appeared "to be sore around the ribs ...and that they should have picked up on this".
She added that they also told her of an occasion when Millie had a bump on her head, and that McCarney had allegedly said "she must have hit her head against the cot ... they were wondering what he must have done to keep her quiet."
Martin and her mother also allegedly told her that having thought over the past weeks other "instances had happened that they should have queried more .... they were blaming themselves for not having checked more on Millie".
Asked by defence QC John McCrudden had it seemed as if mother and daughter hadn't linked what happened to Millie with child abuse, she added: "They seemed to be blaming themselves for not having done so."
During her evidence the social worker was at pains to point out that she "can't stand over" what she was told, as she "wasn't there" to witness any abuse and was only reporting what she was told.
Mrs. Campbell said that after "the terrible news was given that 'Millie was gone' ....both Rachael and Mrs. Graham were very distressed and crying, with Rachael pleading for Millie to be kept on the ventilator.
"Rachael just wanted her to be kept going," said the social worker, who added that "both were crying, saying they could not believe this had happened".
Mr. Tony McHugh, a sub-contractor on the new South West Acute Hospital in Enniskillen, told the court that on the morning of December 10, McCarney arrived for work as normal and "seemed fine". However, later that day he "phoned me that he was sick".
Mr. McHugh said he told McCarney to "hang on for about five minutes" and he would talk to him. He said McCarney "came over to the van and told me he was 'f--king going home'... he seemed to be agitated".
He said McCarney told him he had been sick and "if I didn't believe him to look over where the steel was lying, that he was sick over where the steel was lying".
He told the court that he "didn't want to get into any confrontation" with McCarney and just said "whatever, like".
He also claimed that he and another man had gone over to where McCarney had pointed to look and see if there was any sick "but there was no sign".
Later he told McCarney's defence QC Elis McDermott that swearing on a building site was "not unusual".
He also mentioned that while there were "lumps of muck" lying about, the site was "generally a clean area".
Cross-examined by Martin's QC, Mr. John Crudden, the sub-contractor accepted that he had told police there was "definitely something bothering" McCarney that morning.
He also told police McCarney "appeared to be raring up at someone on the phone" and appeared frustrated and angry.
He claimed McCarney was "in a wild panic to get away...he just wanted to get away in a hurry".
A family friend of Margaret Graham said she saw Millie at her grandmother's house the morning before she was rushed to hospital.
"As soon as I saw her I knew she wasn't well; she was in bad form and her eyes looked puffy like and her right eye was kind of closed, not fully closed, half closed," said Ms Josephine McCaffrey.
Ms McCaffrey, who had called to go shopping with Mrs. Graham, said she also noticed a "wee burn" on Millie's right finger. She said that while she could not remember who mentioned it, she thought she was told the infant burnt it on a candle but that it had been treated.
She said she wasn't sure who commented that Millie "had a wild bad dose (of flu)".
She was asked by Ms McDermott if there was something about the child's eyes which had concerned her. She replied: "Well she looked sick. I thought she might have had a particularly bad flu."
However, she admitted that she had never seen a child's eyes like that before.
Earlier she had told the court she had been at Mrs. Graham's house on November 26, Rachael Martin's birthday, and had "noticed that Millie had a bump on her head...it was in the centre of her forehead...it was like an egg shape".
She said that was the last time she saw Millie.
McCarney, who denies murdering Millie, is also accused of sexually penetrating the infant girl, and with unlawfully and maliciously causing the child grievous bodily harm, and causing her death by an unlawful act.
Martin is accused of failing to protect her baby daughter from the unlawful act that caused her death, and wilfully neglecting her in a manner likely to cause her unnecessary suffering.
The trial continues.