Sometimes got "Unable to reach any database" error


#1

Hello,

I upgrade Phabricator from a old version (April 2016) to last stable :

phabricator 33b77e4abf9ea9abf1ad8f62880f0f140a015008 (Wed, Feb 21)
arcanist 8fe1d7701e5daa3b7c8d96847fa335c0fbf66816 (Fri, Feb 16)
phutil 6d394d4dab71a3c39cb6136ad112c134aa900ac8 (Fri, Feb 16)

Everything works fine but sometimes (not know how to replicate) I got this error

Unable to Reach Any Database. Unable to establish a connection to any database host (while trying “phabricator_phrequent”). All masters and replicas are completely unreachable.

, with button Proceed with Caution. The sql db name change every times, is not related to phabricator_phrequent.
On the bottom-left corner I got the message:

Phabricator is in read-only mode (unreachable master)

It’s enough reload page (or reload 3/4 times) and comes back working fine. It’s quite annoying and worrying.

I don’t know if is a problem of my configuration or is a bug.

Thanks

Paolo


#2

Do you have more than one DB configured? Could the DB be having stability issues?

See https://secure.phabricator.com/book/phabricator/article/cluster_databases/ for how it’s supposed to be configured.


#3

Thanks and sorry for the delay but I was quite busy last days…

I have a single DB hosted on the same machine where is installed Phabricator. I take a look at the mysql error log and I see a lot of messages like this:

[Warning] Too many connections

and sometimes this

`[Warning] Aborted connection 1444046 to db: ‘phabricator_repository’ user: ‘root’ host: ‘localhost’ (Got timeout reading communication packets)

I notice that we have (we are about 15 people using the system) the Unable to Reach Any Database message error when we push some code to repo or add a task comment and refresh the page.

Meanwhile I google mysql message to understand the reason why.

Thanks

Paolo


#4

I see that mysql configuration has as max number of connection 50 https://stackoverflow.com/questions/14331032/mysql-error-1040-too-many-connection.
I will try increase it to 100 and on monday, when everyone use Phabricator, I will see if it fix the problem.

Paolo


#5

You may need increase it to a lot more than 100 - phabricator can open quite a few db connections (Looks like the default value for mysql is 151).


#6

Reading the Max_user_connections from the server variables from our MySQL instance (~50 simultaneous users) says we hit a maximum of 215 connections at some point in the last couple of days (since I last restarted)

However during the import of the repo (conversion of large repo from a legacy VCS) we would hit closer to 800 connections at times.

From my notes, we set our maximum to be 1000 from 151, (no real logic just finger in the wind guess based on the 800 maximum)

Hope that helps


#7

It’s a week that I changed value from 50 to 151 and the error never appear.

Thanks @avivey and everyone