Some 73% of respondents to the Which? poll said they had experienced a faulty public charger at least once in the previous 12 months, while 37% said it was difficult to find a working charger.
Drivers also complained about chargepoints being too expensive and difficult to use.
One Which? member said they could not use a charger because it required an app to be downloaded and their phone had poor signal.
Another described public charging infrastructure as “dismal”, claiming it is “infinitely more complex” to charge an electric car than buy petrol.
In July last year the Government introduced legislation enabling electric car drivers to only need one app to pay for public chargepoints operated by different companies, and to use contactless payments.
Which? head of consumer protection policy Sue Davies said: “We know that people want to make more sustainable choices and switch to an electric vehicle, but many drivers have a lack of confidence in the public charging infrastructure, and this also risks alienating people who aren’t able to charge their EVs at home.
“It is positive that the Government passed new rules that should see improvements in consumers’ experiences of the public charging network.
“The Government and chargepoint operators must continue working together to ensure the UK’s charging infrastructure is up to scratch.
“Charging must be easy and reliable to support more people to make the move to an electric car.”
Which? surveyed 1,004 of its members in November last year who own a pure electric or plug-in hybrid vehicle.
Department for Transport (DfT) figures published on Wednesday show there were 53,677 public electric vehicle charging devices in the UK at the start of this year, an increase of 45% from 12 months earlier.
A DfT spokeswoman said the UK is on track to “install 300,000 public chargepoints by 2030”.
She added: “Our new public chargepoint regulations will ensure simpler pricing information and payment methods, and will require all rapid chargepoints to be 99% reliable on average.
“While we expect the private sector to deliver the majority of chargepoints, we continue to work closely with them to expand and improve the network – with our Zero Emission Vehicle mandate giving the private sector the certainty they need to invest in more chargepoints.”